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Abstract 

This  study  assessed  AFIT's  SYS200  course,  "Acquisition  Planning 
and  Analysis,"  to  determine  whether  it  serves  adequately  as  a  broadening 
course  under  the  Acquisition  Professional  Development  Program  (APDP). 
APDP  certification  requires  persons  in  eight  acquisition  fimctional 
disciplines  (Acquisition  Logistics,  Communications/Computers,  Financial 
Management,  Systems  R&D/Engineering,  Manufacturing,  Quality 
Assurance,  Program  Management,  and  Test  &  Evaluation)  to  complete 
SYS200,  an  intermediate-level  overview  of  the  acquisition  process. 

Three  complementary  measures  (assessing  knowledge,  self-efficacy, 
and  post-course  job  performance)  were  used  to  determine  whether  SYS200 
conveyed  its  curriculum  effectively  to  members  of  all  fimctional  disciplines. 
The  results  showed  that  SYS200  attendance  produced  substantial  increases 
in  acquisition-related  knowledge,  self-efficacy,  and  supervisor  assessments 
for  all  fimctional  disciplines.  In  all  three  measures,  no  significant  differences 
in  increases  between  fimctional  discipUnes  were  foimd.  These  results 
indicate  that  SYS200  is  functioning  well  as  a  broadening  course. 

This  study  also  identified  three  areas  (Risk  Management,  Personal/ 
Leadership  Skills,  and  Sustainment)  which  coiild  enhance  SYS200  course 
performance  relative  to  APDP  goals. 

In  addition,  acquisition  experience  was  found  to  be  a  negligible  factor 
in  predicting  course  performance.  This  supports  the  conclusion  that  the 
APDP  certification  requirement  for  completion  of  SYS200  should  not  be 
waived  based  solely  on  years  of  experience. 
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AN  ASSESSMENT  OF  AFIT  SYS200, 

"ACQUISITION  PLANNING  AND  ANALYSIS," 

AS  A  BROADENING  COURSE  UNDER  APDP 

I.  Introduction 

Of  all  the  continually  changing  disciplines  in  government  and 
management  today,  perhaps  none  is  more  dynamic  than  the  field  of  defense 
systems  acquisition  management.  Spanning  the  boundaries  of  technology, 
finance,  national  security,  and  politics,  acquisition  management  is  arguably 
the  most  challenging  management  field  known  today.  Recognition  of  these 
challenges  has  prompted  several  legal  and  policy  initiatives  in  recent  years. 
These  wide-ranging  initiatives  have  been  designed  (in  part)  to  ensure  that: 

(1)  people  involved  in  acquisition  management  have 
adequate  education  and  training  to  do  their  jobs  properly,  and 

(2)  people  selected  for  senior  acquisition  management 
positions  have  both  the  requisite  knowledge  and 
acquisition-related  experience. 

In  1991,  Congress  passed  Public  Law  101-510,  entitled  the  Defense 
Acquisition  Workforce  Improvement  Act  (DAWIA),  which  expanded  the  scope 
and  definition  of  acquisition  management.  As  a  result,  DAWIA  greatly 
increased  the  number  of  DoD  personnel  who  must  have  acquisition  education 
in  order  to  meet  certification  requirements. 

The  Air  Force's  Acquisition  Professional  Development  Program  (APDP) 
implements  the  training,  education  and  certification  requirements  of  DAWIA 
and  corresponding  DoD-level  implementation  guidance  contained  in  DoD 
5000.52M,  Career  Development  Program  for  Acquisition  Personnel. 
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General  Issue 


APDP  certification  requires  persons  in  eight  of  the  nine  acquisition 
functional  disciplines  as  defined  by  AFR  36-27  (Acquisition  Logistics, 
Communications/Computers,  Comptroller,  Developmental  Engineering, 
Manufacturing/Quality  Assurance,  Program  Management,  Science  & 
Technology,  and  Test  &  Evaluation)  to  complete  an  intermediate  level  course 
in  acquisition  management.  (Currently,  the  sole  exception  to  this 
requirement  is  the  Contracting  fimctional  discipline.) 

The  primary  means  of  fulfilling  this  intermediate  training 
requirement  for  APDP  certification  is  completion  of  SYS200,  "Acquisition 
Planning  and  Analysis,"  offered  by  the  Air  Force  Institute  of  Technology, 
Wright-Patterson  AFB,  OH  (AFR  36-27,  1990). 

Specific  Problem 

SYS200,  which  was  originally  developed  as  an  intermediate-level 
overview  of  the  entire  acquisition  process  targeted  specifically  for  acquisition 
project  managers,  has  not  been  evaluated  to  determine  how  well  it  is 
performing  its  expanded  role  as  a  broadening  course  for  all  APDP  functional 
disciplines. 

Research  Question 

Is  SYS200  course  (as  currently  described  in  the  thirteenth  edition  of 
the  syllabus)  effective  as  an  intermediate-level  overview  of  the  acquisition 
process  for  all  functional  disciplines  which  require  it  for  APDP  certification? 
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Research  Objectives 


(1)  Identify  and  document  the  current  course  content. 

(2)  Measure  the  level  of  increase  in  attendees'  acquisition-related 
knowledge  (pretest/posttest). 

(3)  Measure  the  level  of  increase  in  attendees'  self-efficacy 
regarding  application  of  course  knowledge  and  techniques  in  their 
work  setting  (pretest/posttest). 

(4)  Measure  supervisor  perception  of  increase  in  attendee  knowledge,  self- 
efficacy,  and  work  performance  three  months  after  course  attendance. 

(5)  Identify  differences  in  measures  that  are  attributable  to 
functional  discipline. 

(6)  Identify  evidence  of  broadening  for  all  functional  disciplines. 

Research  Hypotheses 

Hlg:  Differences  in  competency  test  scores  (course  exams)  can  be  attributed 

to  functional  discipline  of  attendees. 

H23:  Differences  in  self-efficacy  scores  can  be  attributed  to  functional 

discipline  of  attendees. 

Differences  in  supervisor  assessments  can  be  attributed  to  hmctional 
discipline  of  attendees. 


Scope 

This  study  does  not  attempt  to  revalidate  APDP  educational 
requirements,  nor  does  it  review  the  curricultim  to  define  adequate 
functional  discipline  knowledge.  Instead,  it  assesses  the  degree  to  which 
SYS200  is  conveying  its  course  content  to  its  newly  expanded  audience  under 
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APDP.  In  analyzing  this  major  objective,  other  relationships  of  interest 
between  measurements  observed  and  student  demographics  are  reported. 

Summary 

This  research  effort  focuses  on  evaluating  the  effectiveness  of  SYS200 
as  a  broadening  course  under  APDP.  The  differences  (or  lack  thereof)  in 
course  and  related  job  performance  measures  between  members  of  the 
various  functional  disciplines  are  used  to  gauge  this  effectiveness. 

In  the  next  section,  acquisition  professional  development  (and  the  roles 
of  APDP  and  SYS200,  specifically)  will  be  explored  in  greater  detail  through 
a  review  of  the  pertinent  literature. 
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II.  Literature  Review 


Introduction 

This  literature  review  traces  the  evolution  of  the  Acquisition 
Professional  Development  Program  and  summarizes  secondary  sources  of 
information  relevant  to  this  research. 

History  of  Acquisition  Professional  Development 

The  concept  of  acquisition  professional  development  is  an  outgrowth  of 
numerous  trends  within  government  and  industry  over  the  past  forty  years. 
This  concept  has  evolved  in  recent  years  to  become  the  dominant  theme  in 
the  management  of  DoD  acquisition  personnel. 

The  primary  factor  which  caused  the  creation  and  influenced  the 
evolution  of  acquisition  professional  development  was  a  multitude  of  defense 
acquisition  program  cost  overruns,  beginning  as  far  back  as  the  1950's. 

These  overruns  led  to  widespread  criticism  of  DoD  management  practices, 
including  the  lack  of  formal  training  and  qualiflcations  for  acquisition 
personnel  (Fox,  1988:42-43). 

The  need  for  training  and  education  in  professional  skills  and 
techniques  is  certainly  not  unique  to  defense  acquisition.  Dr.  W.  Edwards 
Doming,  the  noted  expert  on  management  and  quahty,  has  commented  that 

It  is  not  enough  to  have  good  people  in  your  organization.  They  must 
be  continually  acquiring  the  new  knowledge  and  the  new  skills  that 
are  required  to  deal  with  new  materials  and  new  methods  of 
production.  (Walton,  1986:84) 

The  first  comprehensive  DoD  acquisition-related  policy  which 
addressed  the  need  for  increased  levels  of  managerial  training  and 
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development  was  DoD  Directive  5000.1,  "Acqiiisition  of  Major  Defense 
Systems,"  published  by  Deputy  Secretary  of  Defense  David  Packard  in 
May,  1970.  In  the  memorandum  which  accompanied  the  directive.  Secretary 
Packard  commented  that  "successful  development,  production,  and 
deployment  of  major  defense  systems  are  primarily  dependent  upon 
competent  people,  rational  priorities,  and  clearly  defined  responsibilities" 
[emphasis  added]  (Fox,  1988:44-45). 

Acquisition  Management  Professional  Development  Program  (AMPDP) 

In  the  early  to  mid-1980's,  renewed  allegations  of  exorbitant  prices  and 
flagrant  waste  put  the  defense  acquisition  community  under  intense  public 
scrutiny.  At  the  same  time,  many  experienced  acquisition  personnel  were 
enticed  by  higher-paying  jobs  in  the  then-expanding  defense  industrial 
sector.  These  factors  led  to  recognition  by  senior  DoD  management  of  a 
requirement  for  a  career  development  model  for  acquisition  personnel  similar 
to  the  "gates"  for  pilots  and  other  rated  flying  personnel  (Lohmeyer:21-22), 

In  1985,  General  Lawrence  Skantze,  AFSC  Commander,  appointed  the 
Acquisition  Manager  Career  Development  Task  Force  under  then-Major 
General  Ronald  W.  Yates.  The  task  force’s  two  objectives  were  to  develop  a 
structured  career  development  model  for  acquisition  managers,  and  to 
establish  a  formal  certification  process  designed  to  be  visible  both  to  service 
members  and  the  public  alike.  These  two  objectives  formed  the  foundation 
for  the  Acquisition  Management  Professional  Development  Program 
(AMPDP),  which  was  the  first  program  of  its  kind  in  DoD  (Lohmeyer:23). 
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Acquisition  Reform  Initiatives 

The  evolution  of  acquisition  professional  development  has  been 
influenced  by  several  reform  initiatives  which  have  been  established  by 
policy  or  legislation. 

In  1986,  the  President's  Blue  Ribbon  Commission  on  Defense 
Management  (commonly  referred  to  as  the  Packard  Commission  after  its 
chairman,  former  Deputy  Secretary  of  Defense  Packard)  called  for  "dramatic 
improvements  in  our  management  of  acquisition  personnel  at  all  levels 
within  the  Department  of  Defense"  (Gansler,  1989:207).  The  Packard 
Commission  recommendations  were  echoed  in  similar  initiatives  from  both 
the  legislative  and  executive  branches. 

The  Goldwater-Nichols  DoD  Reorganization  Act  was  passed  later  that 
same  year.  This  law,  in  addition  to  strengthening  the  role  of  the  Chairman 
of  the  Joint  Chiefs  of  Staff  and  creating  the  Vice-Chairman  position,  also 
created  the  position  of  Under  Secretary  of  Defense  for  Acquisition  (USD/A), 
and  realigned  the  Office  of  the  Secretary  of  Defense  (OSD)  so  that  all 
acquisition-related  functions  reported  i  his  individual  instead  of  the 
Secretary  of  Defense  or  the  Deputy  Secretary  of  Defense.  The  law  also 
directed  the  military  services  to  functionally  realign  their  headquarters  to 
reflect  acquisition  management  responsibilities  (Gansler,  1989:137-138). 

At  the  same  time,  (Joldwater-Nichols  sought  to  redefine  the  traditional 
concept  of  acquisition  by  expanding  the  number  of  functions  associated  with 
the  acqtdsition  process.  While  previously  limited  to  "the  management  of 
research  and  development,  procurement,  contracting,  and  contract 
administration,"  the  newly  expanded  definition  included  "logistics. 
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developmental  testing,  ...quality  control,  program  management,  S3rstems 
engineering,  production,  and  manufacturing"  (Fox,  1988:118). 


Defense  Management  Review 

In  1989,  Secretary  of  Defense  Dick  Cheney  chartered  the  Defense 
Management  Review  (DMR)  with  explicit  tasking  to  improve  substantially 
the  performance  of  the  defense  acquisition  system.  The  DMR  reflected  the 
newly  expanded  definition  of  the  acquisition  process,  noting  that 
"approximately  580,000  civilian  and  military  personnel  in  DoD  spend  all  or  a 
subst£intial  part  of  their  workday  in  the  acquisition  field"  (DMR:  12). 

On  the  subject  of  acquisition  professional  development,  the  DMR  was 
explicit: 

Current  arrangements  reflect  a  not  altogether  satisfactory  compromise 
of  two  valid,  but  directly  competing  interests.  On  the  one  hand,  it  is 
undeniably  desirable  that  those  who  manage  the  acquisition  system  be 
highly  attimed,  through  personal  experience  in  the  operational  world, 
to  the  needs  of  military  users.  On  the  other  hand,  if  these  needs  are  to 
be  met  in  the  successful  development  of  major  systems,  it  is 
increasingly  imperative  that  acquisition  managers  possess  a  range  of 
technical  skills  and  a  breadth  of  experience  largely  unavailable  in 
operational  assignments.  It  must  be  recognized  that  attainment  by  a 
military  officer  of  equal  competence  for  senior  field  grade  and  higher 
assignments  in  both  the  operational  and  acquisition  arenas  is 
increasingly  difficult,  and  for  many  purposes  impossible.  New  means 
must  therefore  be  found  to  develop  and  retain  the  variety  of  necessary 
acquisition  skills  in  the  military,  while  at  the  same  time  ensuring  that 
development  of  weapon  systems  reflects  keen  regard  for  operational 
realities.  (DMR:  14) 

Current  Developments 

The  most  recent  legislative  initiative  is  the  Defense  Acquisition 
Workforce  Improvement  Act  (DAWIA).  DAWIA  expands  on  the  basic 
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concepts  found  in  AMPDP,  Goldwater-Nichols,  and  the  DMR,  especially  in 
the  niunbers  and  functional  disciplines  of  people  affected.  DAWIA  also 
requires  a  separate  acquisition  management  structure  in  each  service. 

In  concert  with  DAWIA,  DoD  published  DoD  5000.52M,  entitled 
Career  Development  Program  for  Acquisition  Personnel  in  November  1991. 
DoD  5000.52M  provides  for  the  specialized  education  and  training 
requirements  established  by  the  DMR  and  DAWIA,  as  well  as  combining 
military  and  civilian  acqmsition  education  reqmrements  into  a  single 
volume. 


Acquisition  Professional  Development  Program  (APDP) 

Just  prior  to  the  enactment  of  DAWIA,  the  Air  Force  revamped  the 
existing  AMPDP  program  to  accommodate  the  DMR's  expanded  definition  of 
acquisition  and  increased  certification  requirements.  As  a  result,  the  ciirrent 
Acquisition  Professional  Development  Program  (APDP)  was  established  by 
AFR  36-27  on  26  December  1990.  (In  order  to  be  consistent  with  DAWIA  and 
the  new  DoD  5000.52M,  AFR  36-27  is  already  in  the  process  of  being  revised, 
and  there  is  currently  interim  policy  guidance  from  SAF/AQ  in  effect). 

The  mxilti-faceted  role  of  APDP  is  reflected  in  its  intended  purpose: 

To  maximize  the  professional  development  and  mission 
capability  of  the  acquisition  officer  force  by  setting  forth  a  definitive 
and  viable  professional  development  management  plan  that  produces 
broad-based  managers  capable  of  assuming  middle  and  senior 
management  roles  in  support  of  acquisition.  While  differences  may 
exist  in  precise  routes  established  for  or  followed  by  the  acquisition 
work  force,  there  are  minimum  education,  training,  and  experience 
requirements  that  are  important  for  an  individual's  successful  pursuit 
of  senior  responsibilities.  APDP  also  underscores  the  importance  of 
operational  experience  acquisition  officers  need  to  bring  to  the 
acquisition  process.  (AFR  36-27, 1990:7) 
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APDP  incorporates  all  of  the  requirements  from  the  earlier  AMPDP, 
albeit  in  a  slightly  different  format.  There  are  still  a  number  of  levels  of 
certification  (foxir  for  AMPDP,  three  for  APDP),  spaced  at  roughly  even 
intervals  throughout  a  nominal  20-year  career.  In  addition,  the  three¬ 
pronged  approach  of  "education,  training,  and  experience"  as  the  basis  for 
certification  is  essentially  the  same.  The  biggest  difference  lies  in  the  fact 
that  AMPDP  was  mandatory  for  program  managers  only,  whereas  APDP 
covers  the  entire  spectrum  of  acquisition  activities  (APR  36-27,  1990:6). 

There  are  minor  differences  between  DAWIA,  DoD  5000.52M, 

APR  36-27,  and  APR  40-110  (for  Air  Porce  civilian  acquisition  managers) 
definitions  of  the  acquisition  functional  disciplines.  Table  1  summarizes  the 
categories  used  in  the  various  documents. 

Table  1.  A  Comparison  of  Definitions  of 
Acquisition  Pimctional  Disciplines 

DAWIA  DoD5000.52M  APR  36-27  AFR  40-110 


Program  Mgmt 
Acq  Logistics 
Systems  Planning, 
R&D,  Engineering, 
and  Test 
Procurement 
Quality  Assurance 
Business/Fin  Mgmt 
Property  Mgmt 
Cost  Est/Auditing 
Education/Training 
Construction 


IVogram  Mgmt 
Acq  Logistics 
Science  and 
Engineering 
Procurement 
MfgT^roduction 
Quality  Assurance 
Business/Fin  Mgmt 
Property  Mgmt 
Purchasing 


Program  Mgmt 
Acq  Logistics 
Science  &  Tech 
Dev  Engineering 
Test  &  Evaluation 
Contracting 
Mf^Quality 
Comptroller 
Comm/Computers 


Program  Mgmt 
Acq  Logistics 
Scientist  and 
Engineer 
Contracting  and 
Manufacturing 
Quality  Assurance 
Comptroller 
Comm/Computers 


Despite  the  variations  in  categorization,  the  levels  of  guidance  are  fairly 
consistent.  This  is  because  APDP  was  established  while  DAWIA  was  already 
imder  consideration  by  Congress  (Gardner,  1992).  The  current  interim  policy 
guidance  from  SAP/AQ  more  closely  reflects  the  Congressional  and  DoD-level 
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definitions;  the  changes  to  AFR  36-27  consist  primarily  of  combining  Science 
&  Technology  with  Engineering,  and  separating  Manufacturii^  from  Quality 
Assurance  (SAF/AQ  letter,  1992). 

SYS2QQ.  Course  Syllabus 

APDP  certification  requirements  caused  changes  in  the  target 
audience  for  SYS200.  Whereas  the  focus  of  the  course  had  been  to  provide  an 
intermediate-level  overview  of  the  entire  acquisition  process  for  acquisition 
project  managers,  the  emphasis  shifted  to  providing  a  similar  overview  of  the 
acquisition  process  to  members  of  all  functional  disciplines  (Hill,  1991). 

The  course  content  is  reflected  in  the  SYS200  syllabus  of  instruction, 
which  contains  four  main  parts:  course  objectives,  a  course  description, 
evaluation  criteria,  and  administrative  information.  The  information  below 
is  quoted  from  the  course  syllabus  (which  can  be  found  in  its  entirety  at 
Appendix  A). 

Course  Objectives 

The  purpose  of  the  course  is  "to  prepare  acquisition  functionals  for 
management  level  task  accomplishment  in  systems  acquisition." 

The  specific  objectives  of  the  course  are  for  each  student  to: 

a)  Understand  the  individual  management  processes  involved  in 

systems  acquisition; 

b)  Understand  the  concept  of  an  integrated  team  approach  to 

acquisition;  and 

c)  Develop  /  Refine  skills  in  planning,  analyzing,  executing,  and 

controlling  acquisition  projects. 
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Course  Description 

SYS200  is  organized  according  to  three  main  subject  areas:  planning 
the  program,  achieving  program  objectives,  and  controlling  the  program. 
Twenty-one  individual  lesson  blocks  are  split  among  the  three  subject  areas 
as  shown  in  Table  2. 

Table  2.  Time  Allocation  of  SYS200  Lesson  Blocks 
(1)  Planning  the  program: 


The  Acquisition  Process 

5.5  hrs 

Program  Management 

2.0  hrs 

Acquisition  Planning 

6.0  hrs 

Financial  Management 

7.0  hrs 

Work  Breakdown  Structure 

3.0  hrs 

Scheduling 

4.0  hrs 

Integrative  Planning  Exercise 

13.5  hrs 

(2)  Achieving  program  objectives: 

Systems  Engineering 

4,0  hrs 

Configuration  Management 

2.0  hrs 

Software  Management 

2.0  hrs 

Test  and  Evaluation 

2.0  hrs 

Acqiaisition  Logistics 

4.0  hrs 

Tech  Package  Reqts  Formulation 

6.0  hrs 

Contracting 

2.0  hrs 

Dealing  with  Contractors 

2.0  hrs 

Facilities  Acquisition  Management 

1.5  hrs 

Manufacturing  Management 

2.0  hrs 

(3)  Controlling  the  program: 

Program  Control 

1.0  hrs 

Cost  Estimating 

5.5  hrs 

Performance  Measurement 

7.0  hrs 

[At  least  one  of  these  lesson  blocks  is  directly  applicable  to  every 
functional  discipline  identified  in  the  Acquisition  Professional  Development 
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Program.  Together,  the  lessons  combine  to  cover  the  entire  acqmsition 
process.] 


Evaluation  Criteria 

SYS200  student  evaluation  criteria  and  grade  weights  are  summarized 
in  Table  3. 


Table  3.  SYS200  Evaluation  Criteria 
Evaluation  Area _ %  of  Grade 


Pretest 

N/A 

Quiz  #1 

20 

Quiz  #2 

20 

Quiz  #3 

20 

Integrative  Exercise 

15 

Other  Exercises 

25 

[Because  the  integrative  exercise  and  other  exercises  were  completed 
as  group  efforts,  and  because  these  teams  were  not  organized  by  functional 
discipline,  scores  on  these  exercises  were  not  considered  in  this  research.] 

Department  of  Defense  Program  Management  Body  of  Knowledge  (DBOK) 

In  addition  to  cross-referencing  SYS200  objectives  with  APDP 
requirements,  SYS200  course  content  can  be  compared  to  the  Department  of 
Defense  Program  Management  Body  of  Knowledge  (DBOK)  recently  proposed 
by  Kobylarz  in  1991. 

Kobylarz  proposed  a  defense-specific  project  management  body  of 
knowledge  intended  to  be  "sufficiently  comprehensive  to  include  those  project 
managerial  practices  which  every  professional  [acquisition]  project  manager 
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must  know  and  understand  in  order  to  perform  effectively"  (Kobylarz, 
1991:275). 

At  the  highest  level,  the  proposed  DBOK  contains  the  following  areas: 

Software  Management 
Cost  Management 
Contract  Management 
Management  Techniques 
Defense  Program  Management 
Systems  Engineering 
Strategy  and  Planning 
Risk  Management 
Leadership/Personal  Skills 
Logistics  Management 
Manufacturing  Management 
Quality  Management 
Test  and  Evaluation 


In  a  qualitative  comparison  of  the  second  level  of  detail  in  the  proposed 
DBOK  to  the  SYS200  individual  lesson  objectives,  there  is  consistent 
agreement  in  almost  all  areas.  Of  the  thirteen  areas  identified,  the  only 
areas  which  are  not  covered  in  depth  by  the  SYS200  course  curriculimi  are 
Leadership/Personal  Skills  and  Risk  Management.  The  fact  that  this 
agreement  exists  reflects  favorably  on  the  use  of  SYS200  as  a  cross¬ 
functional  broadening  course. 

Summary 

This  review  has  summarized  the  developments  which  led  to  the 
concept  of  acquisition  professional  development,  culminating  in  the 
establishment  of  APDP.  APDP  and  other  certification  requirements  led  to 
changes  in  the  target  audience  and  focus  of  SYS200,  as  reflected  in  the 
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course  syllabus.  These  changes  are  consistent  with  current  attempts  to 
define  acquisition  educational  objectives. 

The  next  section  describes  the  methodology  used  to  assess  the  course's 
effectiveness  in  providing  a  broad  acquisition  perspective  to  all  fimctional 
disciplines. 
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III.  Methodology 


Introduction 

This  section  discusses  research  design  and  describes  an  explicit 
methodology  for  each  measure  of  course  effectiveness.  Literature  pertaining 
to  the  methodology  is  also  reviewed. 

Research  Design 

According  to  Emory  and  Cooper,  the  most  widely  accepted  classes  of 
experimental  designs  are:  1)  pre-experimental,  2)  true  experimental,  and 
3)  quasi-experimental.  True  experiments  are  the  strongest  and  typically  use 
equivalent  control  groups,  some  form  of  randomized  assignment  to  groups, 
and  both  pretests  and  posttests  to  control  for  threats  to  internal  validity. 
Quasi-experiments  are  the  next  strongest  and  are  typically  used  when 
equivalent  .group  design  is  not  possible.  Pre-experiments  are  the  weakest  of 
the  three  and  are  used  in  the  absence  of  pretests  and/or  control  groups 
(Emory  and  Cooper,  1991:428). 

Designs  should  be  fitted  to  the  research  problem  and  environment 
rather  than  the  other  way  around.  Practical  considerations  in  this  study 
limited  the  ability  to  establish  equivalent  control  groups,  to  select  the  test 
group  at  random,  and  to  use  a  pretest  in  one  measure.  These  limitations  are 
discussed  further  in  the  section  on  internal  validity. 

This  study  used  the  following  modified  pre-experimental  design: 
0X0  Knowledge  Test  Measure 

0X0  Self-Efficacy  Measure 

X  O  Supervisor  Assessment  Measure 
In  this  symbology,  X  represents  the  treatment  (course  attendance),  and 
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O  represents  the  observation  or  data  collection  (either  a  pretest  or  posttest 
depending  in  its  position  relative  to  course  attendance). 

Choice  of  Measures 

Measurements  which  cannot  be  made  directly  must  rely  on 
observations  of  some  indirect  indicator  of  the  variable  of  interest.  For 
example,  while  a  physicist  may  accurately  measure  attributes  such  as 
volume,  velocity,  or  amoimts  of  energy,  the  psychologist  is  interested  in 
measuring  aptitudes,  interests  and/or  achievement.  In  education,  the 
estimate  of  achievement  can  only  be  observed  by  measuring  its  effects 
(Jones,  1971:337). 

Because  evaluation  of  educational  effectiveness  is  by  nature  an 
indirect  measure,  there  is  room  for  interpretation  and  disagreement  as  to 
whether  these  indirect  indicators  accurately  reflect  the  true  measure  being 
sought.  In  situations  such  as  these,  correlation  of  complementary  measures 
lends  strength  to  inferences  being  made.  For  example,  in  Streitmater's  study 
of  Total  Quality  Management  (TQM)  training,  knowledge  test  scores  and  self- 
efficacy  scores  were  used  together  to  provide  a  more  thorough  measure  of 
course  effectiveness  (Streitmater,1991). 

Self-efficacy  refers  to  an  individual's  belief  in  his  or  her  capability  to 
perform  a  specific  task  (Gist,  1987:472).  Accordingly,  self-efficacy  gauges  the 
amoimt  of  knowledge  an  individual  can  actually  apply  in  subsequent  job 
performance,  and  serves  as  an  appropriate  measure  of  course  effectiveness. 

In  addition  to  knowledge  and  self-efficacy  measures  similar  to  those 
used  by  Streitmater,  this  study  surveyed  the  supervisors  of  SYS200 
attendees  to  compare  their  assessments  of  changes  in  attendees'  acquisition 
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knowledge,  job-related  confidence,  and  job  performance.  Correlation  of  the 
results  of  the  three  measures  was  used  to  provide  a  stronger  indication  of  the 
true  measure  of  course  effectiveness. 

Construct  Validity 

The  construct  in  this  study  is  that  increases  in  acquisition-related 
knowledge,  self-efficacy,  and  subsequent  job  performance  are  indicators  of 
SYS200  course  effectiveness,  and  that  equivalent  increases  across  functional 
disciplines  indicate  effectiveness  in  a  broadening  role. 

Construct  validity  refers  to  the  meaningfulness  of  assessing  abstract 
characteristics  with  empirical  measures.  This  is  a  function  of  the  theory 
involved  and  the  measuring  instruments  used.  Construct  validity  is 
"evaluated”  rather  than  "measured"  as  it  is  not  directly  observable  (Emory 
and  Cooper,  1991:182).  Construct  validity  may  be  established  "by  showing 
that  different  measures  of  the  same  construct  are  highly  correlated"  (Gist 
and  Mitchell,  1992:183). 

There  is  evidence  to  support  the  assertion  that  educational 
effectiveness  and  job  performance  are  closely  linked.  Gregory  and  Rao  cite 
several  studies  of  both  the  public  and  private  sectors  (including  research 
conducted  by  the  Hudson  Institute  and  the  Rand  Corporation,  among  others) 
which  have  identified  poor  training  as  a  major  cause  of  poor  performance 
(Gregory  and  Rao,  1990:477). 

Support  also  exists  for  the  use  of  self-efficacy  measures  as  indicators  of 
ability  and  performance.  For  example,  "Bandura  has  found  that  self-efficacy 
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is  strongly  related  to  actual  (^ture)  task  performance"  (Locke  et  al., 
1984:242).  Gist  also  cites  seven  studies  that  have  reported  significant 
correlations  between  self-efficacy  and  subsequent  task  performance 
(Gist,  1987:474). 

These  findings  suggest  that  knowledge,  self-efficacy,  and  actual  job 
performance  are  closely  related  and  are  complementary  in  their  ability  to 
provide  a  measure  of  SYS200  effectiveness. 

The  aspect  of  the  construct  which  assesses  broadening  is  based  on  the 
premise  that  the  SYS200  curriculum  does  address  the  entire  spectrum  of 
acquisition  functional  disciplines,  as  discussed  in  Chapter  II.  In  evaluating 
the  broadening  aspect,  an  equivalent  increase  in  measures  across  hinctional 
disciplines  would  not  by  itself  indicate  that  the  course  is  broadening  -  only 
that  each  group  is  learning  the  course  material  in  an  equivalent  fashion. 

Instead,  the  research  should  demonstrate  that  each  functional 
discipline  group  learns  more  about  other  aspects  of  acquisition  (those  more 
closely  associated  with  the  other  functional  disciplines)  than  about  those 
with  which  it  is  already  familiar.  Of  course,  each  group  might  be  expected  to 
perform  better  than  the  others  on  material  specific  to  its  own  area.  If  the 
course  is  effectively  broadening,  however,  each  group  will  be  exposed  to 
equivalent  levels  of  both  familiar  and  new  material.  In  this  case,  equivalent 
group  performance  would  be  expected. 

If  the  course  were  not  effectively  broadening,  lower  overall 
performance  by  a  functional  discipline  group  would  indicate  that  material 
they  know  well  is  absent  from  the  course  and  that  they  are  facing  new 
material  to  a  greater  degree  than  the  other  groups.  Conversely,  higher 
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overall  performance  by  a  functional  discipline  group  would  indicate  that  they 
are  facing  less  new  material  than  the  other  groups. 

To  gain  further  support  for  this  construct,  an  investigation  was 
conducted  which  compared  functional  discipline  group  performance  on  each 
lesson  block.  The  analysis  compared  the  hmctional  disciplines'  mean 
differences  between  pretest  and  posttest  scores  on  the  self-efficacy  measure, 
with  the  lowest  group  mean  difference  indicating  that  all  the  other  functional 
disciplines  showed  a  higher  increase  on  that  particular  lesson  block.  This 
analysis  was  performed  for  each  lesson  block  that  could  be  considered  most 
relevant  to  a  single  functional  discipline.  The  results  are  shown  in  Table  4 
and  demonstrate  further  support  for  the  construct  used  in  this  study. 

Table  4.  Support  For  Broadening  Construct 


Llfiaa^mBlock 
Financial  Management 
Configuration  Management 
Test  &  Evaluation 
Acquisition  Logistics 
Contracting 

Manufacturing  Management 
Cost  Estimating 


Functional  Discipline  Achieving 
Lowest  Mean  Increase 

Financial  Management 
S&T/Engineering 
Test  &  Evaluation 
Acquisition  Logistics 
Contracting 
Manufacturing/QA 
Financial  Management 


The  only  two  functional  disciplines  that  did  not  show  the  lowest 
increase  on  any  single  lesson  block  were  the  Program  Management  and 
Communications/Computer  groups.  This  does  not  erode  support  for  the 
construct;  rather,  it  indicates  that  these  two  disciplines  tend  to  "span" 
different  functions,  and  no  single  lesson  block  can  be  considered  most 
relevant  to  these  two  disciplines. 
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The  other  two  measures  did  not  provide  similar  opportunities  to  break 
out  results  by  lesson  block  to  further  assess  the  construct. 

Internal  Validity 

Internal  validity  refers  to  the  extent  to  which  the  study  measures  what 
it  was  intended  to  measure  and  that  the  study  results  have  meaning  (Emory 
and  Cooper,  1991:180).  Ebel  provides  essentially  the  same  definition  as 
"intrinsic  validity"  which  "involves  the  use  of  experimental  techniques  other 
than  correlation  with  a  criterion  to  provide  objective,  quantitative  evidence 
that  the  test  is  measuring  what  it  ought  to  measure"  (Ebel,  1972:437). 
Internal  validity  can  also  be  considered  a  hmction  of  design,  measurement 
reliability,  measurement  content  validity,  and  construct  validity 
(Shane,  1991). 

Because  the  pre-experimental  design  used  is  not  very  strong,  a  number 
of  effects  may  have  been  present,  potentially  threatening  internal  validity. 
Examples  of  these  effects  are  maturation,  testing,  instrumentation,  selection, 
and  experiment  mortality  (Emory  and  Cooper,  1991:425). 

Each  measure  used  in  this  study  had  different  strengths  and 
weaknesses  in  the  face  of  these  effects.  Complementary  use  of  mtdtiple 
measures  in  this  study  helped  to  defend  against  these  effects,  thus  mitigating 
the  inherent  weakness  of  the  design. 

Maturation  Effects 

Maturation  refers  to  changes  taking  place  "within  the  subject  which 
are  a  function  of  the  passage  of  time  and  are  not  specific  to  any  particular 
event"  (Emory  and  Cooper,  1991:425). 
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Because  the  time  horizon  of  each  SYS200  course  was  short  (three 
weeks),  maturation  was  discounted  as  a  confotmd  for  the  resiilts  of  the 
knowledge  test  and  self-efficacy  measures.  The  supervisor  assessment, 
however,  could  have  been  affected  since  the  time  horizon  was  longer  (three 
months  between  course  completion  and  administration  of  supervisor  survey). 
However,  the  mean  of  course  attendees'  acquisition-related  experience  was 
fo\md  to  be  6.9  years,  so  three  additional  months  of  "maturation"  wo\ild 
appear  to  be  negligible. 

If  the  mean  experience  level  differed  by  functional  discipline,  then 
maturation  effects  might  also  have  inappropriately  influenced  study 
findings.  An  analysis  of  variance  (ANOVA)  of  Experience  versus  Functional 
Discipline  showed  that  the  null  hypothesis  of  "no  difference"  would  only  be 
rejected  at  the  44%  confidence  level.  This  supports  a  conclusion  that  there 
was  no  statistical  difference  in  experience  level  across  the  functional 
discipline  groups.  (This  ANOVA  and  the  effects  of  experience  on  study 
findings  are  discussed  in  further  detail  imder  Assessment  of  Methodology.) 

Use  of  the  three  complementary  measures  (two  of  which  were  not 
susceptible  to  maturation  effects)  also  helped  to  minimize  maturation  effects 
on  study  findings. 

Testing  Effects 

Testing  effects  refer  to  the  fact  that  "the  mere  experience  of  taking  the 
first  test  can  have  a  learning  effect  that  influences  the  results  on  the  second 
test"  (Emory  and  Cooper,  1991:425).  This  threat  applies  to  the  knowledge 
test  and  self-efficacy  measures.  This  effect  could  be  mitigated  through  the 
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use  of  control  groups;  however,  practical  considerations  outweighed  the  use  of 
a  control  group  for  these  measures. 

The  knowledge  test  measure  actually  consisted  of  evaluating 
performance  on  a  comprehensive  pretest  and  three  quizzes  staggered  through 
the  course.  To  completely  control  for  testing  effects  would  have  required 
three  control  groups  in  addition  to  the  test  group.  This  would  have 
drastically  reduced  the  sample  size  for  testing  purposes  (137  students  across 
nine  functional  areas  further  divided  into  four  groups  would  leave  about  four 
students  per  group),  substantially  reducing  statistical  power  of  the  statistical 
tests  being  used.  Moreover,  all  students  had  to  take  the  quizzes  in  order  to 
pass  the  course,  making  the  use  of  control  groups  infeasible. 

A  similar  sample  size  problem  would  exist  for  the  self-efficacy  measure 
(109  responses  across  nine  functional  groups  divided  into  two  groups  leaves 
six  students  per  group). 

It  is  reasonable  to  assmne  that  testing  effects  would  be  equivalent 
across  all  fimctional  discipline  groups.  Testing  effects  make  it  difficult  to 
detect  the  absolute  magnitude  of  changes  between  pretests  and  posttests  that 
may  be  attributed  to  the  treatment  (attending  the  course).  Since  the  main 
thrust  of  the  research  was  to  assess  broadening,  determining  if  this  change 
occurs  equitably  across  all  fimctional  disciplines  took  precedence  over 
determining  the  absolute  magnitude  of  change. 

Instrumentation  Effects 

Instrumentation  effects  refer  to  reductions  in  internal  validity  that 
result  from  changes  in  the  testing  environment,  changes  to  the  measuring 
instrument,  or  changing  the  observer(s)  (Emory  and  Cooper,  1991:425). 
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The  fact  that  every  supervisor  assessment  in  this  study  was  performed 
by  a  different  observer  is  an  example  of  this  type  of  threat.  The  only  control 
in  this  case  was  to  assume  that  extreme  differences  in  "hard"  versus  "soft" 
raters  balance  out.  Helmstadter  states; 

This  t3rpe  of  error  [rater  bias]  need  be  of  no  great  concern  to  the 
careful  user  of  rating  scales.  Appropriately  motivated  raters  can 
be  helped  to  overcome  such  specific  biases,  and,  since  they  are 
different  for  each  rater,  they  tend  to  cancel  out  when  separately 
obtained  individual  ratings  are  pooled  (Helmstadter,  1964:191). 

While  ratings  were  pooled  in  this  study,  there  were  not  enough  ratings  in 

each  pool  (functional  discipline)  due  to  the  limited  time  duration.  Rater  bias 

remains  a  potential  threat  to  the  validity  of  the  supervisor  survey  measure. 

Since  the  self-efficacy  measure  was  "self-reported,"  it  too  was  subject  to 

similar  rater  bias  in  that  attendees  were  rating  themselves.  In  addition, 

strengths  associated  with  the  scale  terms  used  may  have  different  meanings 

for  different  individuals.  The  pools  were  larger  for  this  measure,  but  were 

substantially  unbalanced  across  functional  disciplines  and  still  too  small  in 

some  functioned  disciplines  to  conclude  that  biases  would  cancel  out. 

The  knowledge  test  measure  was  not  subject  to  rater  bias.  Correlation 

of  the  other  two  measures  with  the  knowledge  measure  helped  to  mitigate 

the  threat  of  rater  bias  to  the  validity  of  the  research  restdts. 

Other  instrumentation  effects  may  also  have  been  present.  For 

example,  a  change  in  knowledge  test  questions  would  represent  such  an 

effect.  For  the  first  class  measured,  the  course  instructor  decided  to  "throw 

out"  two  ambiguous  questions  on  one  of  the  quizzes.  Some  scores  were 

adjusted  by  as  much  as  eight  points  (on  a  100  point  scale).  This  caused  a 

significant  impact,  as  an  ANOVA  of  the  Knowledge  Posttest  Score  (quiz 
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average)  by  Class  showed  a  statistically  significant  difference  between 
classes  at  the  99.4%  confidence  level. 

This  threat  was  mitigated  in  several  ways.  First,  this  adjustment 
occurred  for  the  first  group  under  observation.  Subsequently,  these 
questions  were  reworded  for  the  other  classes  so  that  they  had  a  fairer 
chance  of  answering  correctly  (in  other  words,  their  scores  were  also  adjusted 
implicitly).  Another  mitigating  factor  was  that  the  adjustment  applied  to  a 
single  quiz;  when  averaged  with  the  other  two  quizzes,  the  greatest  impact 
the  adjustment  could  have  had  is  2.6  points  on  the  posttest  score. 

Correlation  of  results  with  the  other  two  measures  (where  no  adjustment 
took  place)  also  helped  to  minimize  this  potential  threat. 

Selection  Effects 

"One  of  the  more  important  threats  to  internal  validity  is  the 
differential  selection  of  subjects  to  be  included  in  experimental  and  control 
groups"  (Emory  and  Cooper,  1991:425). 

Control  groups  were  not  selected  for  practical  reasons  as  previously 
discussed.  Moreover,  random  selection  of  the  class  attendees  could  not  occur 
as  they  were  selected  by  the  existing  Air  Force  education  and  training 
assignment  process  (as  defined  in  AFR  50-5). 

There  were  other  problems  as  well.  Partitioning  the  attendees  would 
have  created  insufficient  sample  sizes.  Pseudo-randomly  assigning  an 
equivalent  control  group  would  have  required  determining  the  demographics 
of  the  test  group,  then  randomly  sampling  that  portion  of  the  acquisition 
workforce  who  had  not  attended  the  course  until  the  makeup  of  the  group 
reflected  that  of  the  test  group.  Both  of  these  approaches  were  infeasible. 
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These  considerations  resulted  in  the  pre-experimental  design  being 
used.  However,  as  stated  previously,  this  research  relied  on  the  use  of 
multiple  complementary  measures  to  strengthen  findings. 

Experiment  Mortality 

"Experiment  mortality  occurs  when  composition  of  the  study  groups 
changes  during  the  test"  (Emory  and  Cooper,  1991:426).  This  study  would 
also  have  experienced  a  similar  effect  on  validity  if  the  course  content 
changed  during  this  research  effort. 

Some  minor  changes  did  occur  during  the  research.  If  a  student 
dropped  out  during  the  course,  observations  from  all  measures  attributable  to 
that  student  were  dropped.  The  course  content  was  not  changed  appreciably 
during  the  study,  but  some  lessons  were  presented  by  different  instructors 
for  some  classes.  In  addition,  as  previously  discussed,  there  were  some 
changes  made  to  the  knowledge  test  between  classes.  In  the  aggregate, 
however,  the  course  content  and  associated  measures  remained  essentially 
the  same.  Therefore,  experiment  mortality  was  not  considered  to  be  a 
significant  threat. 

Reliability  of  Measures 

Emory  defines  a  measure  as  reliable  "to  the  extent  to  which  it  supplies 
consistent  results."  There  are  three  characteristics  of  reliability:  stability, 
equivalence,  and  internal  consistency  (Emory  and  Cooper,  1991:185). 

Stability  refers  to  consistent  results  between  repeated  measures  of  the 
same  person.  This  study  did  not  repeat  measures  on  individuals,  so  this 
characteristic  of  reliability  was  not  evaluated. 
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Equivalence  refers  to  consistency  between  raters.  This  characteristic 
applied  to  the  supervisor  assessment  and  self-efficacy  measures  and  could 
not  be  measimed  as  previously  discussed. 

Internal  consistency  as  a  measure  of  reliability  is  used  when  the 
instnunent  is  used  only  once  on  each  test  subject.  It  was  possible  to  measure 
internal  consistency  for  two  of  the  three  measiires  as  further  discussed  below. 

Internal  Consistency  of  Knowledge  Tests 

Internal  consistency  of  the  knowledge  tests  was  evaluated  using  the 
Norm-Referenced  Differentiation  Index  which  measures  "how  well  the  high 
scorers  on  the  entire  test  did  on  an  item  compared  to  how  well  the  low  scorers 
on  the  entire  test  did  on  [that]  item"  (AFM  50-62,  1984:25-9) . 

This  index  also  evaluates  a  test’s  ability  to  detect  differences  among 
students.  This  index  is  the  ratio  (H-L)/(N/3);  where  H  equals  the  number  of 
students  in  the  high  1/3  group  who  answered  the  item  correctly,  L  equals  the 
number  of  students  in  the  low  1/3  group  who  answered  the  item  correctly,  N 
equals  the  total  number  of  students  who  took  the  test,  and  3  equals  the 
number  of  rank  order  groups  (AFM  50-62,  1984:25-10).  The  average 
Differentiation  Indices  for  the  knowledge  tests  are  presented  in  Table  5. 

The  "Handbook  for  Air  Force  Instructors"  suggests  the  following 
Differentiation  Index  assessment  criteria  (AFM  50-62, 1984:25-10): 

Unacceptable:  -1.00  to  0.00 

Marginally  Acceptable:  0.00  to  0.25 

Probably  Acceptable:  0.25  to  1.00 
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Table  5.  Average  Differentiation  Indices  For  Knowledge  Tests 


January  Pretest 

.24 

January  Quiz  #1 

.21 

January  Quiz  #2 

.20 

January  Quiz  #3 

.20 

March  Pretest 

.25 

March  Quiz  #1 

.20 

March  Quiz  #2 

.22 

March  Quiz  #3 

.22 

April  Pretest 

.25 

April  Quiz  #1 

.21 

April  Quiz  #2 

.19 

April  Quiz  #3 

.20 

Results  of  this  analysis  suggest  that  none  of  the  knowledge  tests  were 
unacceptable  in  internal  consistency  and  their  ability  to  detect  differences 
among  students. 


Internal  Consistency  of  Self-EfScacv  Measure 
Cronbach’s  Coefficient  Alpha  was  used  to  assess  internal  consistency 
of  the  self-efficacy  measure.  This  coefficient  estimates  how  well  scores 
obtained  by  testing  under  a  single  condition  represent  absolute  scores.  This 
coefficient  is  a  ratio  of  the  variance  component  for  persons  to  the  sum  of  the 
variance  components  for  persons  and  residuals,  which  results  in  variance  due 
to  condition  effects  being  factored  out.  In  essence,  this  is  a  ratio  of  "signal"  to 
"signal-plus-noise"  (Cronbach,  1970:156-161). 

Internal  consistency  for  the  self-efficacy  measure  was  evaluated  for 
each  of  the  19  lessons  contained  in  the  course  syllabus.  Results  of  this 
analysis  are  shown  in  Table  6. 
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Table  6.  Self-Efficacy  Measure  Reliability  by  Lesson  Block 

(Cronbach's  Alpha) 


Lesson  Block 

Pretest 

Posttest 

The  Acquisition  Process 

.848 

.910 

Program  Management 

.886 

.894 

Acquisition  Planning 

.916 

.903 

Financial  Management 

.862 

.871 

Work  Breakdown  Structiu’e 

.931 

.920 

Scheduling 

.906 

.879 

Systems  Engineering 

.863 

.882 

Configuration  Management 

.879 

.837 

Software  Acquisition  Mgt 

.913 

.897 

Test  &  Evaluation 

.883 

.795 

Acquisition  Logistics 

.900 

.945 

Tech  Package/Reqts  Formulation 

.914 

.869 

Contracting 

.881 

.848 

Dealing  with  Contractors 

.887 

.836 

Facilities  Acquisition  Mgt 

.890 

.907 

Manufacturing  Management 

.854 

.888 

Program  Control 

.713 

.760 

Cost  Estimating 

.820 

.914 

Performance  Measurement 

.873 

.900 

Steel  suggests  the  following  evaluation  criteria  (Class  notes,  1992): 


Alpha  greater  than  .9 
Alpha  between  .8  aind  .9 
Alpha  between  .7  and  .8 
Alpha  less  than  .7 


Excellent 

Good 

Fair 

Poor 


These  results  indicate  good  to  excellent  internal  consistency  for  the 
self-efficacy  measure. 


Internal  Consistency  of  Supervisor  Survey 
It  was  not  possible  to  evaluate  the  internal  consistency  of  the 
supervisor  assessment  survey  as  it  contained  only  six  questions,  with  only  a 
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single  question  to  evaluate  each  factor.  This  measure  is  therefore  considered 
to  be  the  least  reliable  of  the  three. 

Content  Validity  of  Measmes 

"Content  validity  is  concerned  with  the  adequacy  of  sampling  of  a 
specified  universe  of  content"  (Ebel,  1972:437).  Measures  are  valid  in  content 
to  the  extent  that  they  provide  adequate  coverage  of  the  topic  under  study 
(Emory  and  Cooper,  1991:180).  The  term  "adequate  coverage"  implies 
elements  of  both  depth  and  breadth.  For  example,  the  number  of  questions 
relating  to  a  specific  block  of  instruction  should  be  allocated  consistent  with 
the  time  spent  covering  each  lesson  block.  Additionally,  the  questions  should 
adequately  reflect  the  lesson  objective  and  the  depth  of  the  material  covered 
without  being  consistently  "too  easy"  or  "too  hard." 

For  the  self-efficacy  measure,  the  number  of  questions  allocated  to 
each  lesson  block  was  determined  by  the  following  process: 

1)  For  reliability  purposes,  each  lesson  block  was  allocated  at  least 

two  questions,  and 

2)  For  proportional  coverage,  one  question  was  added  for  each  hour  of 

instruction  in  the  lesson  block. 

In  most  cases,  the  wording  of  the  tasks  to  be  rated  were  taken  directly  from 
the  lesson  objectives  described  in  the  syllabus.  This  measure  is  therefore 
content  vaUd  to  the  extent  that  the  course  instruction  was  consistent  with  the 
syllabus. 

For  the  knowledge  test,  the  number  and  scope  of  the  questions  was 
determined  by  the  SYS200  faculty,  using  similar  objectives  of  "adequate 
coverage"  (Hill,  1991). 
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External  Validity 


In  contrast  to  internal  validity,  external  validity  refers  to  the  extent  to 
which  the  results  of  research  can  be  generalized  to  broader  groups,  different 
settings,  or  across  time.  Internal  validity  is  a  necessary  but  not  sufficient 
condition  for  external  validity,  so  internal  validity  must  take  precedence 
(Emory  and  Cooper,  1991:427-8). 

The  primary  reqmrement  for  external  validity  in  this  study  was  the 
ability  to  generalize  the  results  to  the  population  of  all  students  who  will 
have  completed  this  version  of  SYS200  (thirteenth  edition  of  the  syllabus). 
This  ability  was  assessed  in  terms  of  sample  composition  and  size. 

Sample  Composition 

In  order  to  assess  the  degree  to  which  the  knowledge  measure  sample 
composition  was  representative  of  the  population  of  acquisition  personnel 
who  require  SYS200  for  APDP  certification,  the  sample  percentages  by 
functional  discipline  was  compared  to  the  corresponding  population 
percentages.  Data  for  the  population  were  taken  from  AFMC/DPU 
reqviirements  for  APDP  certification  (Glover,  1992).  Results  from  this 
comparison  are  shown  in  Figure  1. 
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Log  Comp  Mgt  Engr 

Functional  Discipline 

Figure  1,  Comparison  of  SYS200  Sample  and  Population 
by  Functional  Discipline  Percentage 

The  only  notable  discrepancies  are  in  the  Financial  Management 
(Comptroller)  and  Contracting  functional  disciplines.  The  discrepancy  in  the 
Contracting  Functional  Discipline  is  easily  explained  as  SYS200  is  only  a 
"desired"  (i.e. ,  not  required)  course  for  certification.  For  Financial 
Management,  SYS200  is  not  required  until  Level  III  certification  (which 
could  imply  that  the  majority  of  potential  attendees  in  that  functional 
discipline,  being  relatively  new  to  APDP  participation,  have  not  yet  reached 
that  certification  level  and,  hence,  have  not  attended  SYS200);  however,  this 
is  also  true  for  most  of  the  other  functional  disciplines  as  well.  Overall,  the 
sample  is  generally  representative  of  the  population. 
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Sample  Size 

In  the  time  allotted  for  this  study,  SYS200  generated  a  sample  size  of 
137  students  for  the  knowledge  test,  109  for  the  self-efficacy  measure  and  34 
supervisor  assessments.  These  students  were  distributed  in  an  unbalanced 
fashion  over  nine  functional  disciplines,  further  reducing  the  effective  sample 
size. 

The  main  effect  of  this  small  sample  size  on  the  analysis  is  that 
estimates  of  group  (functional  discipline)  sample  variances  tend  to  be  high 
which  masks  the  ability  for  the  ANOVA  technique  used  to  detect  small  group 
differences  that  coiild  be  present.  Because  of  this  small  sample  size,  care 
must  be  exercised  in  generalizing  conclusions  to  a  larger  group.  (The 
procedures  implemented  in  this  study  have  been  documented  for  future 
investigators  to  continue  the  analysis  with  larger  sample  sizes.) 

Knowledge  Test  Measure  Methodology 

Population  Undgr  Sfady 

For  this  measure,  the  population  consists  of  those  persons  who  have 
attended  or  will  attend  the  current  version  of  SYS200.  The  parameters  of 
interest  are  the  population  mean  score  on  a  defense  acquisition  knowledge 
pretest  and  the  population  mean  of  an  average  of  three  SYS200  course 
quizzes  (posttests  administered  on  the  completion  of  each  week  of  the  course). 
These  population  means  cannot  be  observed  directly  and  are  therefore 
estimated  by  the  mean  of  the  sample  consisting  of  those  persons  attending 
the  course  offerings  observed  in  this  study. 
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Scores  from  this  measure  were  tested  for  the  possibility  that  the 
treatment  (SYS200)  does  not  equivalently  affect  persons  from  different 
functional  disciplines. 

The  relevant  test  h3rpotheses  are: 

Hq.*  pi  =  P2  =— =M-9  0*1®  population) 

At  least  one  pj  is  different 

where  pj  is  the  estimated  population  mean  score  of  each  functional  discipline. 

Development  of  Measure 

The  existing  knowledge  pretest  and  three  course  quizzes  developed  by 
the  AFIT  SYS200  faculty  were  used. 

Data  Collection 

There  were  four  SYS200  course  offerings  during  this  research  effort: 

Class  1:  27  January  - 14  February  1992; 

Class  2:  2  March  -  20  March  1992; 

Class  3:  4  May  -  22  May  1992;  and 

Class  4:  8  June  -  26  June  1992. 

The  tests  were  administered  by  AFIT  faculty  in  a  classroom 
environment,  with  students  placing  responses  on  coded  answer  sheets  for 
scoring  by  computer.  A  different  pretest  was  administered  to  Class  4; 
therefore,  no  knowledge  test  data  from  Class  4  was  included  in  the  analysis. 

De.srription  nf  Sample 

Because  the  knowledge  measure  was  mandatory  for  attendees,  it 
provided  the  largest  sample.  Out  of  a  possible  150  attendees,  pretest  scores 
were  available  for  123  attendees,  and  posttest  scores  were  available  for  137. 
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Consequently,  analysis  was  limited  to  a  sample  size  of  123.  Histograms  by 
functional  discipline  and  experience  are  shown  in  Figure  2  and  Figure  3. 


Log  Mgt  Engr 


Figure  2.  Histogram  of  Knowledge  Test  Sample  by  Functional  Discipline 


Years  Of  Acquisition  Experience 

Figure  3.  Histogram  of  Knowledge  Test  Sample  by  Years  of 
Acquisition  Experience 
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Testing  of  the  null  hypothesis  was  performed  using  a  nonparametric 
analysis  of  variance  technique.  This  test  was  performed  three  times:  once  on 
the  pretest  scores;  once  on  the  quiz  average  (posttest)  scores;  and  once  on  the 
difference  (quiz  average  minus  pretest  scores).  The  results  are  presented  in 
Chapter  IV. 

Analysis  of  Variance  (ANOVA) 

Analysis  of  variance  tests  the  hypothesis  that  the  mean  of  the 
dependent  variable  (in  this  case,  knowledge  test  score)  is  the  same  for  each 
level  of  the  factor  in  question  (in  this  case,  functional  discipline)  (SAS, 
1985:114).  This  is  accomplished  by  comparing  the  estimated  across-factor 
variance  (estimated  variance  of  test  scores  across  functional  discipline)  to 
estimated  within-factor  variance  (test  score  variance  within  each  fimctional 
discipline).  If  the  null  hypothesis  were  known  to  be  true,  these  two  variances 
would  be  close  to  each  other  and  the  ratio  of  these  two  variances  (F-statistic) 
would  be  close  to  one  (Devore,  1991:375).  The  greater  this  ratio  becomes,  the 
stronger  the  evidence  points  to  rejecting  the  null  hypothesis  and  concluding 
that  at  least  one  mean  was  different  from  the  others. 

This  parametric  ANOVA  method  requires  that  the  the  treatment 
distributions  are  all  normal  with  the  same  variance  (Devore,  1991:374). 
These  assumptions  cotild  not  be  verified  for  two  of  the  smaller  frinctional 
discipline  samples;  consequently,  the  use  of  a  "distribution-free"  or 
nonparametric  ANOVA  technique  was  required. 
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NonParametric  ANOVA 

Nonparametric  ANOVAs  test  hypotheses  without  making  assiunptions 
about  the  imderlying  distributions  of  the  data.  "Most  nonparametric 
methods  are  based  on  taking  the  ranks  of  a  variable  and  analyzing  these 
ranks  instead  of  the  original  values"  (SAS:19).  In  practice,  this  implies  no 
longer  estimating  the  population  mean  with  a  sampling  mean,  but  instead 
comparing  a  sampling  statistic  of  rank  to  the  expected  value  of  this  sampling 
statistic,  assuming  the  null  hypothesis  was  true  (Devore,  1991:623). 

The  Kruskal-Wallis  nonparametric  ANOVA  subroutine  in  the 
STATISTDC  software  package  was  used  to  perform  these  tests. 

Self-Efficacv  Measure  Methodology 

Population  Under  Study 

For  this  measure,  the  population  consists  of  those  persons  who  have 
attended  or  will  attend  the  current  version  of  SYS200.  The  parameter  of 
interest  is  the  theoretical  population  mean  rank  of  self-efficacy  survey  scores. 

Restatement  of  Hypotheses 

As  with  the  knowledge  measure,  the  results  from  this  measure  were 
tested  for  the  possibility  that  the  treatment  (SYS200)  does  not  equivalently 
affect  persons  from  different  functional  disciplines. 

The  relevant  test  h3rpotheses  are: 

H,,:  m=|i2=—=1^9  (i-e.,  one  population) 

H^:  At  least  one  pj  is  different 

where  pj  is  the  theoretical  population  mean  rank  of  survey  scores  for  each 
functional  discipline. 
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In  meeting  the  research  objective  of  assessing  the  effectiveness  of 
SYS200  as  a  broadening  course,  the  variable  of  interest  for  this  measure  was 
the  students'  perceived  ability  to  apply  the  knowledge  gained  from  the  course 
to  various  acquisition-related  activities  in  their  normal  duty  assignments. 

Instrument  Design 

The  instrument  developed  for  this  measure  was  a  questionnaire 
designed  to  assess  self-efficacy  in  each  of  the  acquisition  management 
functions  addressed  in  the  SYS200  course  syllabus.  The  questionnaire  asked 
students  to  rate  their  own  abilities  to  accomplish  specific  tasks  pertaining  to 
the  nineteen  acquisition  functions  covered  in  the  SYS200  lesson  blocks. 

One  of  the  obstacles  encoimtered  in  actually  writing  the  questionnaire 
was  balancing  the  requirements  for  comprehensive  content  and  reasonable 
length.  In  general,  the  number  of  specific  tasks  rated  for  each  acquisition 
function  was  determined  by  the  relative  length  and  number  of  major 
objectives  of  each  lesson  block  in  the  syllabus,  while  the  total  number  of 
questions  was  kept  as  low  as  possible  to  avoid  "survey  fatigue." 

In  drafting  the  questions,  the  tasks  to  be  rated  were  taken  directly 
from  the  lesson  objectives  described  in  the  syllabus.  The  actual  number  of 
questions  per  lesson  block  is  shown  in  Table  7. 
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Table  7.  Question  Allocation  per  Lesson  Block 


Lesson  Block  Number  of  Questions 

1  The  Acquisition  Process  7 

2  Program  Management  3 

3  Acquisition  Planning  5 

4  Financial  Management  7 

5  Work  Breakdown  Structure  5 

6  Scheduling  5 

7  Systems  Engineering  5 

8  Configuration  Management  3 

9  Software  Management  4 

10  Test  and  Evaluation  3 

11  Acqtoisition  Logistics  5 

12  Tech  Package  Requirements  Formulation  7 

13  Contracting  4 

14  Deahng  with  Contractors  4 

15  Facilities  Acquisition  Management  3 

16  Manufacturing  Management  3 

17  Program  Control  2 

18  Cost  Estimating  5 

19  Performance  Measurement  4 

Total  84 


In  addition,  nine  questions  pertaining  to  APDP  were  included  to  assess 
some  of  the  "broadening"  aspects  of  acquisition  not  directly  covered  by 
SYS200  (such  as  knowledge  of  roles  and  responsibilities  of  other  fimctional 
disciplines),  and  six  more  were  added  to  assess  perceptions  of  the  APDP 
certification  process.  Student  perceptions  of  the  certification  process  are 
indirectly  tied  to  the  research  objectives  since  they  may  indicate  differences 
in  group  perceptions  of  applicability  of  the  certification  process  to  their 
fimctional  discipline.  Finally,  two  demographic  questions  to  identify 
students  by  functional  discipline  and  years  of  experience  were  added,  for  a 
total  of  101  questions. 
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The  questions  pertaining  to  each  lesson  block  were  grouped  in  order  to 
minimize  shifts  in  subject  matter  and  frame  of  reference  (Emory  and  Cooper, 
1991:372),  and  the  resulting  groups  were  arranged  in  random  order  on  the 
pretest  survey.  To  further  control  for  possible  testing  effects,  the  question 
groups  on  the  posttest  were  placed  in  reverse  order  from  the  pretest 
arrangement  (Babbitt  and  Nystrom,  1989:95). 

Scaling 

A  five-point  Likert  scale  was  used  to  rate  the  levels  of  self-efficacy. 

The  questions  were  phrased  so  as  to  state  ability  to  accomplish  various 
acquisition-related  tasks  and/or  to  apply  certain  acquisition  knowledge  in  a 
given  setting.  The  students  were  then  asked  to  assess  their  agreement  with 
the  statements,  using  the  scale  as  described  below: 

a.  "Do  not  agree"  indicating  the  lowest  level  of  efficacy, 

b.  "Agree  slightly"  indicating  the  next  level, 

c.  "Agree  moderately"  indicating  the  next  level, 

d.  "Agree  considerably"  indicating  the  next  level,  and 

e.  "Agree  completely"  indicating  the  highest  level  of  efficacy. 

The  first  consideration  in  selection  of  this  particular  scale  concerned 
the  use  of  a  unipolar  scale  (as  opposed  to  a  bipolar  scale  in  which  levels  of 
agreement  and  disagreement  lie  on  opposite  ends,  with  a  neutral  midpoint). 
Efficacy,  defined  as  the  ability  to  accomplish  a  certain  task,  can  be  thought  of 
as  complete  (maximum  efficacy),  present  in  some  degree  (moderate  efficacy), 
or  totally  lacking  (zero  efficacy).  Because  there  is  no  such  thing  as  "negative 
efficacy,"  a  unipolar  scale  was  more  appropriate  for  this  questionnaire. 

The  second  consideration  in  scale  selection  was  the  choice  of  nr  lifiers 
to  designate  points  on  the  continuum  of  agreement.  The  modifying  adverbs 
chosen  (Slightly,  Moderately,  and  Considerably)  were  foimd  to  represent  the 
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three  points  between  "none"  and  "completely"  which  roost  closely 
approximate  equal  semantic  intervals  (Babbitt  and  Nystrom,  1989:142). 

Instrument  Refinement 

For  screening  purposes,  the  questionnaire  was  administered  to  thirty 
AFIT  graduate  students  familiar  with  the  SYS200  course  material.  In 
addition  to  identifying  eleven  potentially  problematic  questions,  their 
comments  regarding  clarity,  content  validity,  and  ease  of  administration 
were  invaluable  in  refining  the  survey  instrument. 

Based  on  the  results  of  the  screening,  the  estimated  time  for 
completion  of  the  questionnaire  was  found  to  be  reasonable  (approximately 
twenty  to  twenty-five  minutes),  although  the  screeners’  consensus  was  that 
the  length  of  the  questionnaire  (ten  pages)  was  somewhat  imposing. 
However,  part  of  the  required  length  was  actually  due  to  repeating  the  scale 
responses  at  the  top  of  each  page,  which  received  favorable  comments. 

Data  Collection 

Because  of  time  hmitations  for  survey  administration  in  the  SYS200 
class  schedule,  the  pretest  questionnaires  were  distributed  along  with  the 
rest  of  the  class  paperwork  during  the  first  day.  Students  were  then 
requested  to  complete  the  questionnaire  at  some  time  prior  to  the  next  class 
session  on  the  following  day,  although  questionnaires  were  occasionally 
received  on  the  third  class  day  as  well.  The  posttest  survey  was 
administered  similarly;  students  were  requested  to  complete  and  return  the 
questionnaires  after  attending  the  final  lesson  block. 

A  problem  encoimtered  during  the  first  data  collection  was  a  low 
response  rate,  especially  during  the  posttest  (approximately  thirty  percent). 
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This  was  attributed  primarily  a  prominent  statement  of  "volimtary 
participation"  on  the  questionnaire  instruction  sheet.  This  problem  was 
resolved  during  subsequent  data  collections  by  de-emphasizing  the  voluntary 
nature  of  the  questionnaire.  As  a  result,  the  response  rate  improved 
significantly  to  approximately  seventy  percent  in  later  collections. 

(Both  pretest  and  posttest  questionnaires  can  be  found  at  Appendix  B.) 

Description  of  Sample 

The  self-efficacy  measure  sample  size  suffered,  partly  due  to  its 
voluntary  nature  and  also  partly  due  to  the  reqioirement  for  matched  pretest 
and  posttest  responses.  As  a  result,  it  provided  a  sample  of  109  out  of  200 
possible  (54.5  percent).  Fortimately,  this  smaller  sample  contained  attendees 
from  all  of  the  fimctional  disciplines.  Figures  4  and  5  show  the  composition 
of  the  self-efficacy  measure  sample  by  functional  discipHne  and  by  years  of 
experience,  respectively. 


Log  Mgl  Engr 

Figure  4.  Histogram  of  Self-Efficacy  Sample  by  Functional  Discipline 
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Years  Of  Acquisition  Experience 

Figure  5.  Histogram  of  Self-Efficacy  Sample  by  Years  of 
Acquisition  Experience 

Data  Analysis 

The  data  associated  with  this  measure  is  ordinal  ("Agree 
Considerably"  is  stronger  than  "Agree  Moderately,"  but  the  interval  between 
these  two  is  not  necessarily  equal  to  an  interval  between  two  other  adjacent 
points  on  the  same  scale).  "When  only  the  rank  order  of  scores  is  known, 
means  and  standard  deviations  found  on  the  scores  themselves  are  in  error  to 
the  extent  that  the  successive  intervals  on  the  scale  are  not  equal"  (Siegel, 
1956:26).  For  this  reason,  testing  of  the  stated  null  hypothesis  was 
performed  using  a  nonparametric  analysis  of  variance  technique. 
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Supervisor  Assessment  Measure  Methodology 


Population  Under  Study 

For  this  measure,  the  population  consists  of  those  persons  who  have 
attended  or  will  attend  the  current  version  of  SYS200.  The  parameter  of 
interest  is  the  theoretical  population  mean  rank  of  supervisor  survey  scores. 

Restatement  of  Hypotheses 

As  with  the  other  measures,  the  results  from  this  measure  will  be 
tested  for  the  possibility  that  the  treatment  (SYS200)  does  not  equivalently 
affect  persons  from  different  functional  disciplines. 

The  relevant  test  hypotheses  are: 

Hq:  M-1  =  M-2  =*“=I^9  (i-c. .  one  population) 

Hg.-  At  least  one  Pi  is  different 

where  Pi  is  the  theoretical  population  mean  rank  of  survey  scores  for  each 
functional  discipline. 

Development  of  Measure 

This  measure  was  intended  to  assess  supervisors'  perceptions  of 
students'  acquisition-related  knowledge,  efficacy,  and  performance  after 
completion  of  SYS200.  As  in  the  knowledge  and  self-efficacy  measures, 
increases  in  survey  scores  provide  indirect  evidence  of  course  effectiveness. 

Instrument  Design 

Based  on  prior  AFIT  experience  with  low  response  rates  for  post-course 
supervisor  surveys,  the  primary  considerations  in  the  development  of  this 
questionnaire  were  brevity  and  ease  of  use.  Consequently,  this  questionnaire 
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was  limited  to  one  page  in  length,  and  concentrated  on  whether  supervisors 
perceived  increases  in  students'  acquisition-related  knowledge,  efiBcacy,  and 
performance. 

Because  of  the  initial  success  and  promising  results  from  the  self- 
efficacy  questionnaire,  the  same  unipolar  Likert  scale  was  used.  (The 
supervisor  assessment  questionnaire  can  be  found  at  Appendix  C.) 

Data  Collection 

The  questionnaires  were  mailed  to  the  supervisors  of  SYS200  students 
from  the  January  and  March  class  offerings  approximately  three  months 
after  course  completion.  This  timing  was  intended  to  allow  for  sufficient 
observation  of  post-course  performance,  while  keeping  the  necessary  time  for 
collection  efforts  and  subsequent  analysis  within  the  constraints  of  this 
research  effort. 

Description  of  Sample 

The  supervisor  assessment  had  the  smallest  sample  (60  out  of  100 
possible),  mainly  due  to  data  collection  time  constraints.  Figures  6  and  7 
show  the  composition  of  the  sample  by  functional  discipline  and  by  years  of 
experience,  respectively. 
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Figure  6.  Histogram  of  Supervisor  Assessment  Sample 
by  Subordinates'  Functional  Discipline 


Years  Of  Acquisition  Experience 

Figure  7.  Histogram  of  Supervisor  Assessment  Sample 
by  Subordinates'  Years  of  Acquisition  Experience 
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Data  Analysis 

Testing  of  the  stated  null  h3rpothesis  was  performed  using  a 
nonparametric  ANOVA.  However,  because  of  limitations  due  to  sample  size, 
additional  evaluation  of  survey  results  was  also  required,  consisting 
primarily  of  descriptive  statistics  and  a  summary  assessment  of  supervisor 
comments. 

Realignment  of  Functional  Disciplines 

As  discussed  in  Chapter  2,  SAF/AQ  published  interim  policy  guidance 
in  February  1992  which  superseded  the  functional  discipline  definitions  in 
AFR  36-27  (which  was  used  to  categorize  the  data  for  all  three  measures).  To 
accomodate  the  interim  guidance,  the  Developmental  Engineering  and 
Science  &  Technology  groups  were  combined  for  analysis  purposes  on  all 
three  measures. 

Unfortunately,  complying  with  the  other  realignment  of  functional 
disciplines  --  separating  Manufacturing  from  Quality  Assurance  --  was 
infeasible  for  two  reasons:  in  some  cases,  making  a  distinction  between  the 
two  new  disciplines  was  not  possible  (for  example,  "Chief,  Manufacturing/ 
Quality  Assurance  Division");  moreover,  division  of  the  already  small  sample 
would  have  resulted  in  two  even  smaller  samples  insufficient  in  size  for 
ANOVA  purposes. 

Correlation  of  Measures 

As  previously  discussed,  the  three  complementary  measures  were  used 
to  hedge  against  specific  threats  to  validity,  with  correlation  analysis 
performed  on  the  results  of  the  measures  to  provide  an  indication  of  study 
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validity.  This  assessment  was  based  upon:  1)  a  quantitative  measure  of  the 
correlation  of  the  rank  (based  on  score)  of  each  functional  disipline  on  one 
measure  compared  to  the  rank  of  each  functional  discipline  on  the  other 
measures;  and  2)  a  qualitative  comparison  of  the  consistency  of  conclusions 
supported  by  each  measure. 

Quantitative  rnrrelation  Analysis 

The  Pearson  product  moment  correlation  coefficient  is  "a  stimmary 
measure  of  the  linear  relationship  between  the  paired  values  of  two  random 
variables"  (Kachigan,  1991:130).  The  method  used  to  arrive  at  this 
coefficient  is  essentially  an  "average"  value  of  the  products  of  paired  standard 
scores  for  each  variable.  The  range  of  this  coefficient  is  from  -1  to  +1,  with  -1 
indicating  a  perfect  inverse  linear  relationship  between  the  two  variables,  0 
indicating  no  relationship,  and  +1  indicating  a  perfect  positive  linear 
relationship  (Kachigan,  1991:126). 

The  data  in  this  study  were  not  able  to  meet  all  assumptions  required 
in  order  to  use  Pearson's  methodology.  The  self-efficacy  and  supervisor 
assessment  measures  were  not  measured  on  an  interval  or  ratio  scale;  they 
were  measured  on  an  ordinal  scale  for  which  Pearson’s  method  is  not 
appropriate  (Kachigan,  1991:128). 

The  Spearman  rank  correlation  coefficient  is  more  appropriate  for 
ordinally  scaled  data.  It  is  calculated  in  the  same  fashion  as  Pearson's 
product  moment,  but  uses  ranked  scores  instead  of  the  original  data 
(Statistix,  1991:104).  The  Statistix  software  package  was  used  to  calculate 
the  Spearman  Rank  Correlation  coefficient  for  each  possible  pairing  of  the 
three  measures. 
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This  analysis  was  accomplished  simply  by  observing  whether  each 
measure  supported  the  same  conclusion  and  by  considering  the  reUability  of 
each  measure.  As  discussed  previously,  consistency  between  measures 
enhances  the  validity  of  the  research  results. 

Assessment  of  Methodology 

As  discussed  earlier,  a  combination  of  knowledge,  self-efficacy,  and 
supervisory  assessment  measures  was  used  to  provide  a  more  thorough 
evaluation  of  SYS200  course  effectiveness  than  any  single  measure  could 
have  provided  alone.  Because  the  three  measures  consisted  of  different 
levels  of  data,  different  analysis  techniques  were  considered  for  each 
measure.  For  example,  the  interval  level  data  from  knowledge  measure 
might  have  been  appropriate  for  parametric  analysis  (except  for  the  problem 
of  meeting  the  necessary  underlying  assumptions),  while  the  ordinal  level 
data  from  the  other  two  measures  were  hmited  to  nonparametric  analysis 
techniques. 

Because  the  research  question  for  each  measure  focused  on  the  single 
issue  of  whether  any  functional  discipUne  performed  differently,  and  due  to 
the  limitations  discussed  above,  nonparametric  single  factor  ("one-way") 
ANOVA  was  selected  as  the  most  appropriate  form  of  analysis  for  all  three 
measures.  Use  of  these  ANOVAs  offered  the  advantage  of  consistency  of 
analysis,  which  allowed  for  more  uniform  comparisons  of  results,  as  well  as 
providing  for  construct  validation  through  correlation  of  measures. 
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In  comparing  the  various  functional  disciplines'  performances  through 
single  factor  ANOVA,  the  most  obvious  and  logical  potential  "second  factor" 
lay  in  variance  due  to  different  levels  of  acquisition  experience.  An  initial 
assessment  of  the  available  data  revealed  that  Analysis  of  Covariance  would 
be  ill-advised  because  of  insufficient  sample  composition  (t.e.,  too  many 
empty  cells  in  the  matrix  of  functional  disciplines  and  years  of  experience). 

Subsequent  investigation  revealed  that  the  effect  of  experience  on  the 
variables  of  interest  (i.e. ,  increases  in  acquisition  knowledge,  self-efficacy, 
and  job  performance)  was  actually  negligible;  therefore,  "one-way"  ANOVAs 
by  functional  discipline  would  not  be  adversely  affected  by  discounting  the 
effect  of  experience  as  a  potential  second  factor. 

The  effect  of  experience  was  investigated  by  means  of  two  separate 
analyses.  The  first  consisted  of  comparing  the  experience  levels  of  the 
various  functional  disciplines  through  an  ANOVA.  The  resulting  p-value  of 
.5561  implies  that  there  was  no  significant  difference  in  experience  levels  by 
functional  discipline,  and  shows  support  for  the  proposition  that  all  groups 
were  essentially  equivalent  in  levels  of  experience.  The  range  of  experience 
by  functional  discipline  is  shown  in  Figure  8. 
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Figure  8.  Scatterplot  of  Years  of  Acquisition  Experience 
by  Functional  Discipline 


As  shown,  the  distributions  of  experience  for  each  functional  discipline 
overlap  and  cannot  be  considered  statistically  different. 

The  second  analysis  was  a  linear  regression  of  knowledge  pretest  and 
posttest  scores  to  determine  whether  experience  was  a  significant  factor.  The 
results  showed  that  for  the  pretest,  experience  was  in  fact  a  statistically 
significant  factor,  but  its  coefficient  in  the  regression  equation  (indicating  the 
slope  of  the  regression  line)  was  relatively  small  at  .87,  with  a  corresponding 
r2  of  .1127.  These  figures  imply  that  while  experience  was  a  discemable 
factor,  its  actual  effect  on  differences  in  scores  was  low  (i.e.,  less  than  one 
point  per  each  additional  year  of  experience);  moreover,  only  11  percent  of 
the  variance  in  scores  was  "explained"  or  accounted  for  by  the  variance  in 
experience.  On  the  posttest,  the  regression  showed  that  experience  was  even 
less  of  a  factor,  with  a  coefficient  of  .24  and  a  r^  of  .0181. 
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Because  the  results  of  the  regression  suggested  that  experience  was  a 
negligible  factor  in  determining  differences  in  knowledge  scores,  single-factor 
ANOVAs  were  deemed  sufficient  for  data  analysis.  Any  lingering  doubt 
concerning  levels  of  experience  as  a  potentially  complicating  factor  was 
ultimately  removed  by  the  absence  of  ambiguity  in  the  research  findings:  all 
of  the  research  hypotheses  were  either  strongly  rejected  or  strongly 
supported. 

Summary 

The  methodology  used  in  this  research  incorporated  three 
complementary  measures  to  obtain  an  indirect  measurement  of  the  variable 
of  interest  (SYS200  course  effectiveness).  The  research  design  was  somewhat 
limited  by  environmental  and  practical  considerations,  but  is  representative 
of  similar  research  efforts.  As  discussed,  each  measure  has  its  own  strengths 
and  weaknesses  in  the  face  of  threats  to  internal  validity.  Use  of  multiple 
measures  provides  a  defense  against  undesired  effects  and  lends  strength  to 
the  research  findings.  While  the  results  are  considered  valid  for  the  sample 
taken,  the  small  sample  size  limits  generalizing  results  to  the  larger 
poptdation  of  interest. 

In  the  next  chapter,  the  results  of  the  study  and  subsequent  data 
analysis  are  presented  and  discussed. 
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IV.  Analysis  and  Findings 


Introduction 

This  chapter  discusses  the  application  of  the  methodology  described  in 
Chapter  III  and  presents  results  from  the  three  measures  and  their 
associated  data  analysis  procedmes. 

Knowledge  Measure  Analysis 

Analysis  of  Pretest  and  Posttest 

Figure  9  and  Figure  10  show  histogr  ams  of  knowledge  pretest  and 
posttest  scores,  respectively.  These  figures  show  that  course  attendance 
generated  a  substantial  increase  in  knowledge  test  scores:  the  overall  mean 
for  the  pretest  was  41.9  on  a  scale  of  0  to  100,  and  the  overall  mean  for  the 
posttest  was  82.9  on  the  same  scale. 

In  addition,  the  variability  of  scores  was  lower  on  the  posttest,  as 
indicated  by  the  tighter  grouping  of  scores  aroxmd  the  mean;  pretest  scores 
ranged  from  a  low  of  14  to  a  high  of  66,  while  posttest  scores  varied  from  a 
low  of  50  to  a  high  of  98. 

Figures  11  and  12  show  the  spread  of  pretest  and  posttest  scores  by 
functional  discipline.  (Although  some  differences  in  range  can  be  seen,  it 
cannot  be  determined  from  the  scatterplots  alone  whether  these  differences 
between  functional  disciplines  are  statistically  significant.) 
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Figure  9.  Histogram  of  Knowledge  Pretest  Scores 
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Figure  10.  Histogram  of  Knowledge  Posttest  (Quiz  Average)  Scores 
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Figure  11.  Scatterplot  of  Knowledge  Pretest  Scores 
by  Functional  Discipline 
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Figure  12.  Scatterplot  of  Knowledge  Posttest  Scores 
by  Functional  Discipline 


The  nonparametric  Kruskal -Wallis  ANOVA  test  for  the  knowledge 
pretest  versus  functional  discipline  returned  a  p-vaiue  of  .6305.  The  p-value 
represents  the  smallest  possible  amount  of  type  I  error  (rejecting  the  null 
hypothesis  when  it  was  true).  In  this  case,  .6305  represents  a  high 
probability  of  falsely  rejecting  the  null  hypothesis  that  the  means  are  equal, 
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which  implies  that  the  differences  between  functional  discipline  group  scores 
were  not  statistically  significant.  This  lends  support  to  the  proposition  that 
performance  on  the  pretest  was  essentially  equivalent  across  all  fimctional 
disciplines.  (The  complete  ANOVA  tables  are  located  at  Appendix  D.) 

The  Kruskal-Wallis  ANOVA  test  for  the  knowledge  posttest  scores 
versus  functional  discipline  returned  a  p-value  of  .0272.  This  represents  a 
very  small  probability  of  falsely  rejecting  the  null  hypothesis  (implying  that 
the  differences  between  functional  discipline  group  scores  were  statistically 
significant),  and  lends  strong  support  to  the  proposition  that  performance  on 
the  course  quizzes  was  not  equivalent  across  all  groups. 


Investigation  of  Group  Mean  Differences 

Further  investigation  was  required  to  determine  which  groups  were 
significantly  different.  Table  8  stimmarizes  performance  by  functional 
discipline  groups. 


Table  8.  Mean  Ranks  and  Mean  Scores  of  Knowledge  Posttest 
by  Functional  Discipline 


Functional  Discipline 

MEAN 

BANK 

MEAN 

SCORE 

SAMPLE 

SIZE 

Comm/Computers 

87.3 

86.47 

5 

Test  &  Evaluation 

85.8 

85.95 

19 

S&T/Engineering 

81.8 

85.77 

22 

Other 

81.3 

84.70 

9 

Manufacturing/QA 

70.1 

83.48 

7 

Program  Mgt 

68.2 

83.08 

34 

Financial  Mgt 

51.0 

78.63 

10 

Acq  Logistics 

50.5 

79.12 

26 

Contracting 

44.2 

78.13 

5 

Overall 

69.0 

82.90 

137 
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As  discussed  in  Chapter  III,  the  reason  for  using  "distrihution-free" 
nonparametric  methods  in  this  study  was  that  some  group  scores  could  not  be 
considered  normally  distributed,  or  were  ordinal  measures  for  which 
parametric  methods  would  be  inappropriate.  Analysis  of  the  knowledge 
posttest,  however,  was  an  instance  in  which  the  necessary  assumptions  for 
using  parametric  methods  could  be  met;  using  parametric  ANOVA  offered 
the  advantage  of  being  able  to  identify  those  functional  disciplines  for  which 
the  difference  between  the  group  mean  and  the  overall  mean  was  significant. 

The  use  of  parametric  ANOVA  requires  the  assumption  that  the 
knowledge  scores  for  each  functional  discipline  group  are  normally 
distributed.  Wilk-Shapiro  statistics  assessing  the  normality  of  the  sample  for 
each  group  are  shown  in  Table  9. 


Table  9.  Wilk-Shapiro  Statistics  by  Functional  Discipline 


Acquisition  Logistics  .8168 

Comm/Computers  .9876 

Financial  Management  .9511 

Contracting  .9930 

Manufacturing/QA  .9324 

Program  Management  .9708 

S&T/Engineering  .9297 

Test  &  Evaluation  .9423 


Wilk-Shapiro  statistics  may  be  considered  good  if  higher  than  .8,  and 
excellent  if  higher  than  ,9  (Reynolds,1991).  Therefore,  these  statistics 
reasonably  support  the  assumption  that  the  group  scores  are  normally 
distributed  for  this  measure  (although  the  lower  figure  for  Acquisition 
Logistics  may  indicate  a  sample  with  a  weaker  central  tendency,  which  is 
discussed  in  greater  detail  below). 
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Having  fulfilled  the  requirement  for  meeting  the  assumption  of 
normality,  Tukey's  Procedure  for  multiple  pairwise  comparisons  of  means 
was  used  to  provide  an  indication  of  which  groups  performed  differently  on 
the  knowledge  posttest.  The  results  showed  that  the  Acquisition  Logistics 
group  was  statistically  different  from  both  the  Science  &  Technology/ 
Engineering  group  and  the  Test  &  Evaluation  group;  all  other  pairwise 
comparisons  of  groups  were  considered  homogenous.  (The  complete  Tukey's 
Procedure  output  from  Statistix  is  contained  in  Appendix  D.) 

One  possible  explanation  for  this  difference  is  the  fact  that  the 
Acquisition  Logistics  group  was  actually  comprised  of  two  different  groups: 
those  performing  sustainment-oriented  functions  at  Air  Logistics  Centers, 
and  those  performing  systems  acquisition  logistics  functions  at  Product 
Centers.  Because  the  two  groups  perform  different  functions,  they  were 
reassigned  to  separate  functional  discipline  groups  for  retest. 

Resiilts  of  this  retest  showed  that  the  sustainment-oriented  Logistics 
group  was  statistically  different,  while  the  group  comprised  of  only  those  in 
acquisition  logistics  frinctions  at  Product  Centers  was  statistically 
indistinguishable  from  the  rest  of  the  functional  disciplines.  Possible 
explanations  for  this  finding  are  discussed  in  Chapter  V. 

Difference  Between  Pretest  and  Posttest 

The  principal  construct  in  this  study  is  that  improvements  in 
knowledge,  self-efficacy,  and  performance  indirectly  measure  SYS200 
effectiveness;  and  that  equivalent  improvements  across  functional  disciplines 
represent  effectiveness  in  a  broadening  role.  Therefore,  the  ANOVAs  on 
observed  changes  represent  the  most  important  tests  performed  in  this  study. 
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Figure  13  shows  the  histogram  of  increases  in  knowledge  test 
performance  (essentially,  the  shift  that  was  shown  in  Figures  9  and  10),  and 
Figure  14  shows  the  spread  of  increases  by  ftmctional  discipline. 
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Figure  13.  Histogram  of  Acquisition  Knowledge  Score  Increases 

(Posttest  minus  Pretest) 
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Figure  14.  Scatterplot  of  Acquisition  Knowledge  Score 
Increases  by  Fimctional  Discipline 


The  ANOVA  test  for  the  difference  between  knowledge  pretest  and 
posttest  scores  returned  a  p-value  of  .5159.  This  represents  a  moderately 
high  probability  of  falsely  rejecting  the  null  hypothesis  (implying  that  the 
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differences  were  not  statistically  significant),  and  supports  the  proposition 
that  increases  in  knowledge  were  essentially  equivalent  across  groups. 

Explanation  of  ANQYA  Rgsidts 

An  investigation  was  conducted  to  determine  how  the  p-values  from 
the  three  ANOVAs  (pretest,  posttest,  and  increase)  could  report  that  there 
was  a  significant  difference  for  the  posttest  scores,  but  not  for  the  pretest 
scores  or  the  resulting  increases.  Figure  15  depicts  the  results. 
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Figure  15.  Graph  of  Mean  Knowledge  Score  Increase  from  Pretest  to  Posttest 

by  Functional  Discipline 
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As  shown  in  Figure  15,  the  "clustering"  of  posttest  scores  into  two  distinct 
groups  was  responsible  for  the  significant  p-value  reported;  however,  the 
range  of  posttest  scores  (difference  between  minimum  and  maximum)  was 
almost  exactly  the  same  as  the  same  differential  for  the  pretest  scores.  In 
addition,  the  slopes  of  the  lines  connecting  the  pretest  and  posttest  group 
means  (representing  the  increase)  were  all  essentially  equivalent,  which 
explains  the  insignificant  p-value  reported  by  the  ANOVA  on  the  increase. 

Self-Efficacv  Measure  Analysis 

Analysis  of  Pretest  and  Posttest 

Figures  16  and  17  show  histograms  of  self-efficacy  pretest  and  posttest 
scores,  respectively.  In  contrast  to  the  knowledge  test  score  scale  of  0  to  100, 
the  scale  of  self-efficacy  scores  was  0  to  336  (84  questions  using  a  zero  to  four 
scale).  These  histograms  show  that  course  attendance  generated  a 
substantial  increase  in  self-efficacy  scores:  the  overall  mean  of  pretest  scores 
was  136  out  of  336  (40.5%),  while  the  overall  mean  of  posttest  scores  was  235 
out  of  336  (70%). 

The  range  of  pretest  scores  went  from  a  low  of  17  to  a  high  of  309;  on 
the  posttest,  the  low  was  64,  and  the  high  was  333. 
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Figure  16.  Histogram  of  Self-Efficacy  Pretest  Scores 
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Figure  17.  Histogram  of  Self-Efficacy  Posttest  Scores 
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Figures  18  and  19  show  the  spread  of  scores  by  functional  discipline. 
It  is  not  obvious  from  these  scatterplots  alone  whether  any  differences 
between  functional  disciplines  scores  are  significant. 
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Figure  18.  Scatterplot  of  Self-Efficacy  Pretest  Scores 
by  Functional  Discipline 
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Figure  19.  Scatterplot  of  Self-Efficacy  Posttest  Scores 
by  Fimctional  Discipline 


The  nonparametric  Kruskal-Wallis  ANOVA  test  of  the  self-efficacy 
pretest  versus  functional  discipline  returned  a  p-value  of  .4682.  This 
represents  a  fairly  large  probability  of  falsely  rejecting  the  null  hypothesis 
and  supports  the  proposition  that  pre-course  self-efficacy  relative  to 
performing  acquisition-related  tasks  was  equivalent  for  all  functional 
disciplines. 
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The  nonparametric  Kruskal-Wallis  ANOVA  test  performed  on  the  self- 
efficacy  posttest  score  versus  functional  discipline  returned  a  p-value  of 
.0334.  This  represents  a  very  small  probabihty  of  falsely  rejecting  the  null 
hypothesis  and  strongly  supports  the  proposition  that  post-course  self- 
efficacy  relative  to  performing  acquisition  related  tasks  was  in  fact  different 
among  functional  disciplines. 

Based  on  the  differentiation  seen  in  the  Acquisition  Logistics  group  on 
the  knowledge  measure,  a  similar  breakout  was  performed  for  retest.  The 
results  showed  that,  as  before,  the  sustainment-oriented  Logistics  group 
scored  lower  than  the  remainder  of  the  Acquisition  Logistics  functional 
discipline;  in  this  measure,  however,  the  difference  between  the  sustainment- 
oriented  Logistics  group  rank  and  the  overall  mean  rank  was  not  statistically 
significant. 

Difference  Between  Pretest  and  Posttest 

Again,  the  principal  construct  of  this  study  is  that  improvements  in 
knowledge,  self-efficacy,  and  performeince  indirectly  measure  SYS200 
effectiveness,  and  that  equivalent  improvements  across  functional  discipline 
represent  effectiveness  in  a  broadening  role.  As  before,  the  ANOVAs  for 
observed  changes  represent  the  most  important  tests  performed  in  this  study. 

In  general,  the  increases  in  self-efficacy  scores  mirror  those  seen  in  the 
knowledge  measure  (although  there  was  one  notable  exception,  discussed  in 
greater  detail  under  Correlation  of  Measures).  Figure  20  shows  the 
histogram  of  self-efficacy  increases,  while  Figure  21  depicts  the  range  of 
increases  by  functional  discipline. 
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Figure  20.  Histogram  of  Increases  in  Self-Efficacy  Scores 
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Figure  21.  Scatterplot  of  Increase  in  Self-Efficacy  Scores 
by  Functional  Discipline 

The  nonparametric  Kruskal-Wallis  ANOVA  test  for  difference  in 
posttest  scores  minus  pretest  scores  returned  a  p-value  of  .9626.  This 
represents  an  extremely  high  probability  of  falsely  rejecting  the  null 
hypothesis  and  strongly  supports  the  proposition  that  increases  in 
acquisition-related  self-efficacy  were  essentially  equivalent  across  groups. 
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As  was  done  for  the  knowledge  measure,  an  investigation  was 
conducted  to  determine  how  the  p-value  for  the  posttest  ANOVA  could  be 
reported  as  significant,  while  the  p-values  for  the  pretest  and  increase  were 
not  significEint.  Figure  22  illustrates  the  results. 
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Figure  22.  Graph  of  Mean  Self-Efficacy  Score  Increase 
from  Pretest  to  Posttest  by  Functional  Disciphne 


As  shown  in  Figure  22,  there  was  some  "clustering"  of  posttest  scores, 
which  caused  the  significant  p-value  to  be  reported;  however,  as  was  also 
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seen  in  the  knowledge  measure,  the  range  of  posttest  scores  (difference 
between  minimum  and  maximum)  was  essentially  the  same  as  the 
differential  for  the  pretest  scores.  In  addition,  the  slopes  of  the  lines 
connecting  the  pretest  and  posttest  group  means  (representing  the  increases 
in  self-efficacy)  were  again  essentially  equivalent,  which  explains  the 
insignificant  p-value  reported  by  the  ANOVA  on  the  increase. 

Supervisor  Survey  Measure  Analysis 

As  described  in  Chapter  III,  this  measure  consisted  of  questionnaires 
which  accompanied  the  traditional  AFIT  course  critiques  mailed  out  after 
course  completion.  Supervisors  were  asked  to  assess  the  strength  of  any 
improvement  in  knowledge,  job  task-related  confidence  (efficacy),  and  job 
performance. 

In  general,  the  small  functional  discipline  sample  sizes  were  not 
conducive  to  ANOVA  procedures.  Consequently,  the  majority  of  the  analysis 
and  evaluation  for  this  measure  was  done  subjectively,  consisting  primarily 
of  descriptive  statistics  and  a  summary  assessment  of  supervisor  comments. 
The  results  were  then  used  to  provide  affirmation  of  the  findings  of  the  other 
two  measures. 

Description  of  Survey  Results 

Because  the  survey  was  designed  to  gauge  improvement  since  SYS200 
attendance.  Question  1  was  used  to  assess  the  validity  of  the  supervisor's 
assessment  by  confirming  that  the  subordinate  had  in  fact  worked  for  the 
same  supervisor  both  before  and  after  course  attendance.  Those  surveys 
responding  negatively  to  Question  1  were  not  included  in  subsequent 
analysis. 
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Question  2  was  analogous  to  the  pretest  in  the  knowledge  measure. 
The  histogram  of  responses  is  shown  in  Figure  23. 


Do  Noi  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  Coostderably  Completely 

Figure  23.  Histogram  of  responses  to  "Before  Attending  SYS200,  My 
Subordinate's  Knowledge  Of  Acquisition  Management  Was  Satisfactory. " 


Question  3  was  analogous  to  the  difference  between  pretest  and 
posttest  on  the  knowledge  measure.  Figure  24  shows  the  responses. 


Do  Not  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  Considerably  CompleMly 

Figure  24.  Histogram  of  responses  to  "Since  Attending  SYS200,  My 
Subordinate's  Knowledge  Of  Acquisition  Management  Has  Improved." 


In  general,  these  responses  are  consistent  with  the  results  of  the 
knowledge  measure. 
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Question  4  was  analogous  to  the  increase  observed  in  the  self-efficacy 


measure.  Figure  25  shows  the  responses. 


Do  Not  Affoe  Agree  Apee  Agree 

Agree  Slightly  Modeniely  Ccuiderebly  CooipieiBly 

Figure  25.  Histogram  of  responses  to  "Since  Attending  SYS200,  My 
Subordinate's  Job-Related  Confidence  Has  Improved. " 

These  responses  echo  the  results  of  the  self-efficacy  measure  in  similar 
fashion  to  the  parallels  between  the  previous  question  and  the  knowledge 
measure. 

Figure  26  shows  the  responses  to  Question  5,  which  assessed 
improvement  in  job  performance  since  SYS200  attendance. 


Do  Not  Agroe  Agrve  Apve  Agree 

Agree  Slightly  Moderetely  Ccnsiderably  Corepiecely 

Figure  26.  Histogr  am  of  responses  to  the  question  "Since  Attending  SYS200, 
My  Subordinate's  Current  Job  Performance  Has  Improved. " 
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Figure  27  shows  the  responses  to  Question  6,  which  assessed  course 


relevance  to  attendee's  current  duties. 


Do  Not  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  ConsideiaMy  Completely 

Figure  27.  Histogram  of  responses  to  "Since  Attending  SYS200,  My 
Subordinate  Has  Been  Able  To  Use  The  Information 
Presented  In  The  Course." 


In  general,  the  supervisor  survey  responses  show  moderate  support  for 
the  research  construct,  as  well  as  consistency  with  the  other  two  measures. 

Summary  of  Supervisor  Comments 

Supervisors  were  also  asked  to  provide  comments  or  suggestions  to 
assist  in  course  improvement.  Of  the  60  surveys  returned,  21  included 
comments  (contained  in  Appendix  E).  Table  10  provides  a  summary. 


Table  10.  Categorization  of  Supervisor  Comments 


Category  Frequency 

Experience  Waiver  7 

Currictilum  5 

Too  Soon  to  Comment  3 

Faculty/Facilities  2 

Course  Length  2 

Class  Size  1 

Course  Availability  1 
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For  the  comments  categorized  tmder  "Experience  Waiver,"  three 
directly  stated  that  experienced  persons  shotdd  not  be  required  to  take  the 
course;  two  stated  the  course  was  a  "square  filler";  and  two  said  the  course 
was  beneficial  even  though  their  subordinate  had  substantial  experience. 

For  the  comments  relating  to  curriculum  breadth,  one  was  favorable 
across  the  board;  two  asked  for  enhancements  to  areas  specific  to  their 
subordinate's  functional  discipline  (one  engineering  and  one  logistics);  and 
two  stated  that  the  course  material  was  not  directly  applicable  to  their 
subordinates'  daily  activities  (one  a  program  element  monitor,  and  one  a  test 
and  evaluation  engineer). 

Three  supervisors  thought  that  the  span  of  two  to  three  months  was 
too  soon  to  assess  the  course's  effect  on  their  subordinates,  but  two  expressed 
the  view  that  the  course  would  eventually  have  a  positive  effect. 

In  the  faculty/facilities  category,  one  comment  was  unfavorable  to  a 
single  instructor  in  a  single  lesson  block,  and  one  comment  stated  that  the 
satellite  video  link  was  distracting  and  hindered  student  participation.  Both 
comments  relative  to  course  length  stated  that  the  comse  was  too  long  for 
their  subordinates  to  be  away  from  their  main  duties. 

A  supervisor  whose  subordinate  was  in  the  Financial  Management 
functional  discipline  asked  that  the  course  be  made  more  available.  This 
comment  is  consistent  with  the  Chapter  III  analysis  of  course  composition, 
where  it  was  shown  that  the  sample  contained  a  percentage  of  students  from 
Financial  Management  which  was  significantly  less  than  the  population  of 
those  who  require  SYS200. 
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Correlation  Of  Measures 


Figure  28  shows  that  the  sample  for  all  three  measures  are  similar  in 
composition,  which  mitigates  the  concern  that  uneven  composition  could 
affect  the  degree  to  which  measures  support  each  other. 


35 


30  + 


^  Knowledge 
I  Efficacy 
H  Supervisor 
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Acq  Comm/  Fin  Contr  M/QA  PM  S&T/  T&E  Other 
Log  Comp  Mgt  Engr 


Figure  28.  Comparison  of  Sample  Composition 


As  already  shown,  the  results  from  the  three  measures  paralleled  each 
other  fairly  well.  This  was  particularly  true  for  the  knowledge  and  self- 
efficacy  measures,  which  had  sufficient  sample  size  to  run  ANOVAs.  The 
ANOVAs  were  highly  consistent  in  terms  of  significance  of  results,  as  well  as 
rank  ordering  of  functional  disciplines.  This  consistency  was  further 
investigated  both  by  subjective  analysis  and  by  the  Spearman  rank 
correlation  procedure. 

As  previously  seen  in  Figure  15  and  Figure  22  (which  illustrated  the 
mean  increases  in  knowledge  and  self-efficacy  by  functional  discipline),  the 
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overall  results  of  the  two  measures  were  very  similar  in  nature.  Table  11 
compares  the  rank  orders  of  fimctional  disciplines  on  the  pretests  and 
posttests. 


Table  11.  Rank  Orders  of  Functional  Disciplines 


Rank 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 


Knowledge  Pretest 
S&T/Eng 
T&E 

Acq  Logistics 

Mfg/QA 

Other 

Program  Mgt 
Contracting 
Comm/Computers 
Financial  Mgt 
Logistics 


Knowledge  Posttest 
Comm/Computers 
T&E 
S&T/Eng 
Other 
Mfg/QA 
Program  Mgt 
Acq  Logistics 
Financial  Mgt 
Contracting 
Logistics 


Rank  Sel£Efficacv  Pretest  Self-Efficacv  Posttest 

1  Program  Mgt  Program  Mgt 

2  Financial  Mgt  Financial  Mgt 

3  Contracting  Mfg/QA 

4  T&E  S&T/Eng 

5  S&T/Eng  T&E 

6  Logistics  Contracting 

7  Mfg/QA  Acq  Logistics 

8  Acq  Logistics  Other 

9  Other  Logistics 

10  Comm/Computers  Comm/Computers 


As  shown,  the  posttest  ranks  are  generally  consistent  with  their 
corresponding  pretest  ranks  for  both  measures.  In  addition,  the  ranks  from 
each  measure  are  fairly  consistent  with  each  other.  However,  as  mentioned 
previously,  there  is  at  least  one  notable  exception:  the  Comm/Computers 
fimctional  discipline  turned  in  the  highest  mean  knowledge  score,  while 
consistently  scoring  at  the  bottom  in  the  self-efficacy  measure.  (It  is  also 


73 


interesting  to  note  that  Program  Management  and  Financial  Management 
were  ranked  the  highest  in  mean  self-efficacy,  but  were  in  the  middle  of  the 
pack  in  mean  knowledge  scores.)  One  possible  explanation  for  the 
Comm/Computers  discrepancy  is  discussed  in  Chapter  V. 

The  mean  scores  were  then  analyzed  using  the  Spearman  rank 
correlation  procedure.  Table  12  lists  the  scores  used  in  this  analysis. 


Table  12.  Mean  Scores  by  Fimctional  Discipline 


Fimctional 

Pretests 

Posttests 

lacTfiase 

-  S-E 

S-E 

_3=K 

Acq  Log 

43.53 

124.8 

81.56 

212.5 

38.10 

87.7 

C/C 

38.33 

91.3 

86.47 

185.7 

48.44 

94.3 

FinMgt 

36.33 

152.8 

78.63 

248.8 

42.15 

96.0 

Contr 

38.67 

137.0 

78.13 

227.8 

36.11 

90.8 

Mfg/QA 

41.71 

126.5 

83.48 

233.8 

41.76 

107.3 

PgmMgt 

40.66 

153.4 

83.08 

261.3 

42.44 

107.9 

S&T/Eng 

45.26 

132.8 

85.77 

232.6 

40.47 

99.8 

T&E 

44.00 

135.3 

85.95 

228.7 

41.67 

93.4 

Other 

40.67 

102.6 

84.70 

212.3 

44.04 

109.7 

Logistics 

35.50 

129.0 

73.63 

208.8 

38.13 

79.8 

Because  the  focus  of  the  research  was  on  comparing  the  increases  in 
acquisition  knowledge,  self-efficacy,  and  job  performance  by  functional 
disciplines,  correlations  were  calculated  for  the  increases  shown  above.  The 
Spearman  procedure  returned  a  correlation  coefficient  of  .70  for  the  results  of 
the  knowledge  and  self-efficacy  measures. 

For  criteria  in  assessing  correlation,  "a  reasonable  rule  of  thumb  is  to 
say  that  the  correlation  is  weak  if  0<  I  r  I  <.5,  strong  if  .8<  I  r  I  <1,  and  moderate 
otherwise"  (Devore,  1991:489).  Therefore,  the  two  measures  are  moderately 
correlated,  showing  good  support  for  study  validity. 
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Although  the  supervisor  survey  measure  did  not  produce  meaningful 
ANOVA  ranks  to  compare  to  the  other  two  measures,  an  average  score  from 
the  three  questions  assessing  increases  in  knowledge,  confidence,  and  job 
performance  was  calculated  and  correlated  with  the  knowledge  measure 
using  the  same  Spearman  ranking  procedure.  The  results  showed  a 
correlation  coefficient  of  .62  for  the  knowledge  increase  and  the  supervisors' 
survey  composite  increase,  again  showing  moderate  correlation  between  the 
two  measures. 

A  sensitivity  analysis  was  performed  to  assess  the  chaise  in 
correlation  from  the  omission  of  the  Comm/Computers  functional  discipline 
(which  had  been  the  obvious  outlier  in  the  previous  comparison  by  ranks). 
The  new  Spearman  rank  correlation  coefficient  for  the  knowledge  and  self- 
efficacy  measures  was  found  to  be  .85,  which  would  indicate  high  correlation 
of  the  two  measures  (allowing  for  the  Comm/Computers  exception). 

Histograms  of  Additional  APDP-Related  Questions 

As  discussed  in  Chapter  III,  additional  APDP-related  questions  were 
added  to  the  self-efficacy  questionnaire,  in  part  to  assess  attendees' 
knowledge  and  opinions  of  APDP  and  also  to  supplement  the  assessment  of 
SYS200's  broadening  role.  The  first  group  of  questions  assessed  the  degree 
to  which  an  individual  imderstands  the  roles  and  responsibilities  of  other 
functional  disciplines.  For  these  questions,  a  shift  to  the  right  in  strength  of 
response  would  be  expected  (for  those  individuals  who  attended  the  coxirse). 
Histograms  of  the  pretest  and  posttest  responses  to  these  questions  are 
presented  below  and  show  the  expected  shift. 
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Figure  29  depicts  the  responses  to  the  question  regarding  the 
Acquisition  Logistics  functional  discipline. 


35  -r 


Do  Not  Apoe  Apoe  Apce  A^ee 

Agree  Slightly  Modenteiy  Considertbly  Completely 


Figure  29.  Histogram  of  responses  to  ”I  Can  Describe  The  Roles  & 
Responsibilities  of  the  Acquisition  Logistics  Functional  Discipline." 


Figure  30  depicts  the  responses  to  the  question  regarding  the 
Communications/Computers  functional  discipline. 


□  PlWM 
■  PoMM 


Do  Not  Afree  Aptt  A(m  A(m 

Aftce  Sliflnly  Moderuely  Comideribly  Completely 


Figure  30.  Histogram  of  responses  to  "I  Can  Describe  The  Acquisition  Roles 
And  Responsibilities  Of  The  Communications/Computers 
Fimctional  Discipline." 
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Figure  31  depicts  the  responses  to  the  question  regarding  the 
Comptroller  (Financial  Management)  functional  discipline. 


□  Pralast 
I  Posttast 


Do  Not  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


Figure  31.  Histogram  of  responses  to  "I  Can  Describe  The  Acquisition  Roles 
And  Responsibilities  Of  The  Comptroller  Functional  Discipline." 


Figure  32  depicts  the  responses  to  the  question  regarding  the 
Contracting  functional  discipline. 


Do  Not  Afree  A^to  Apte  Apet 

Apm  Stightly  Modemely  Consiiknbly  Compkiely 


Figure  32.  Histogram  of  responses  to  "I  Can  Describe  The  Acquisition  Roles 
And  Responsibilities  Of  T3ie  Contracting  Functional  Discipline." 
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Figure  33  depicts  the  responses  to  the  question  regarding  the 
Developmental  Engineering  functional  discipline. 


Do  Not  Apee  Agr«e  Agree  Apee 

Agree  Slightly  Modentely  Considenbly  Completely 


Figure  33.  Histogram  of  responses  to  "I  Can  Describe  The  Acquisition  Roles 
And  Responsibilities  Of  The  Developmental  Engineering 
Functional  Discipline." 

Figure  34  depicts  the  responses  to  the  question  regarding  the 
Manufacturing/Quality  Assurance  functional  discipline. 


Figure  34.  Histogram  of  responses  to  "I  Can  Describe  The  Acquisition  Roles 
And  Responsibilities  Of  The  Manufacturing/Quality  Assurance 

Functional  Discipline." 
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Figure  35  depicts  the  responses  to  the  question  regarding  the  Program 
Management  functional  discipline. 


Do  Not  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


Figure  35.  Histogram  of  responses  to  "I  Can  Describe  The  Acquisition  Roles 
And  Responsibilities  Of  The  Program  Management  Functional  Discipline." 


Figure  36  depicts  the  responses  to  the  question  regarding  the  Science 
&  Technology  functional  discipline. 


Figure  36.  Histogram  of  responses  to  "I  Can  Describe  The  Acquisition  Roles 
And  Responsibilities  Of  The  Science  &  Technology  Functional  Discipline." 


79 


Figure  37  depicts  the  responses  to  the  question  regarding  the  Test  & 
Evaluation  functional  discipline. 


Figure  37.  Histogram  of  responses  to  "I  Can  Describe  The  Acquisition  Roles 
And  Responsibilities  Of  The  Test  &  Evaluation  Functional  Discipline," 


ANOVAs  revealed  no  significant  differences  by  fimctional  discipline 
which  would  indicate  that  the  overall  shift  for  any  particular  group  was 
consistently  low.  As  was  found  in  the  self-efficacy  measure,  however,  the 
Communications/Computers  group  was  ranked  lowest  on  both  pre-course  and 
post-course  surveys,  although  the  differences  were  not  statistically 
significant. 

In  general,  all  functional  disciplines  showed  a  noticeable  increase  in 
awareness  of  other  groups’  acquisition  roles  and  responsibilities.  Table  13 
summarizes  the  results  shown  in  Figures  29  through  37. 
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Table  13.  Shift  in  Median  Response  to  Questions  Regarding 
Acquisition  Roles  &  Responsibilities  from  Pretest  to  Posttest 


Fimctional 

Pretest 

Posttest 

Acq  Logistics 

1 

2 

Comm/Computers 

0 

2 

Comptroller 

1 

2 

Contracting 

1 

3 

Engineering 

1 

2 

Mfg/QA 

1 

2 

Program  Mgt 

2 

3 

S&T 

1 

2 

T&E 

1 

2 

While  these  results  show  that  awareness  increased  across  the  board, 
the  differences  shown  between  functional  roles  are  partly  due  to  the  sample 
composition  (for  example,  Program  Management  had  the  most  attendees,  and 
Comm/Computers  had  the  least,  which  affected  the  median  response  to  the 
questions  pertaining  to  those  ftmctional  disciplines).  Nevertheless,  these 
figures  still  indicate  that,  in  the  aggregate,  knowledge  of  other  functional 
disciplines  is  increasing,  providing  additional  evidence  that  SYS200  is 
effective  in  a  broadening  role. 

Questions  Regarding  Perceptions  of  APDP 
Other  questions  were  related  to  attendees’  perceptions  of  their 
opportunity  and  ability  to  meet  certification  requirements  for  their  ftmctional 
discipline  imder  the  APDP  program.  Since  these  were  not  specifically 
addressed  in  the  course,  changes  in  response  between  pretest  and  posttest 
were  expected  to  be  less  pronoimced  and  less  meaningful.  Of  more  interest  is 
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the  location  of  the  strenf  n  of  response  to  these  questions.  Histograms  of 
pretest  and  posttest  responses  to  these  questions  are  presented  below. 
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Figure  38.  Histogram  of  responses  "I  Can  Describe  The  Purpose  Of  The 
Acquisition  Professional  Development  Program." 


These  responses  imply  that  SYS200  attendees  are  being  exposed  to 
APDP  and  its  intended  purpose. 


Do  Not  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  Considerably  Compleiely 


Figure  39.  Histogram  of  Responses  to  "I  Could  Lis^  My  Functional  Discipline 
Certification  Requirements  Under  APDP." 
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Do  Not  Apee  Apte  Agree  Aptc 

Agree  Slightly  Modenldy  Considerably  Completely 


Figure  40.  Histogram  of  responses  to  "I  Believe  These  Certification 
Requirements  Are  Appropriate  For  Acquisition  Managers  in  My 

Fimctional  Discipline." 


These  responses  show  that  in  general,  SYS200  attendees  are  gaining  a 
slight  increase  in  specific  knowledge  of  APDP  certification  requirements,  and 
that  most  believe  these  requirements  are  appropriate. 


Do  Not  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


Figure  41.  Histogram  of  responses  to  "I  Believe  These  Certification 
Reqturements  Are  Achievable." 
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Do  Not  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  Gxuidenbly  Completely 


Figure  42.  Histogram  of  responses  to  "I  Believe  I  Will  Have  Adequate 
Opportimities  For  Achieving  These  Certification  Requirements." 

These  responses  show  a  general  consensus  that  APDP  certification 
requirements  are  achievable,  and  that  opportunities  for  meeting  these 
requirements  will  be  available. 
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Do  Not  Agree  Agree  Agree  Agree 

Agree  Slightly  Moderately  Considenbly  Completely 

Figure  43.  Histogram  of  responses  to  "SYS200  Should  Be  A  Required  Course 
For  Acquisition  Managers  In  My  Functional  Discipline." 

These  responses  show  that  most  attendees  believe  that  SYS200 
attendance  is  valuable  to  their  functional  discipline. 
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Summary 

The  three  measures  provided  consistent  support  for  the  construct,  as 
well  as  showing  consistent  research  results.  In  all  three  analyses,  no 
significant  differences  in  increases  were  found  among  hmctional  discipline 
groups. 

Indications  of  increases  in  acquisition-related  knowledge,  efficacy,  and 
performance  were  found,  in  addition  to  evidence  of  broadening  for  all 
functional  disciplines.  The  additional  survey  data  regarding  perceptions  of 
APDP  revealed  a  broad  awareness  of  the  program,  as  well  as  general  support 
for  its  goals  and  requirements. 

Conclusions  drawn  from  these  results  are  discussed  in  Chapter  V. 
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V.  Concluaiona  and  Rennmmenrifltinna 


Introduction 

The  knowledge,  self-efficacy  and  supervisor  assessment  measures 
(along  with  a  qualitative  comparison  of  the  SYS200  syllabus  to  Kobylarz's 
proposed  DoD  Program  Management  Body  of  Knowledge)  provided  a 
comprehensive  assessment  of  SYS200's  performance  in  a  broadening  role. 
This  chapter  discusses  the  results  presented  in  Chapter  IV,  beginning  with 
accomplishment  of  research  objectives.  Other  conclusions  are  also  presented 
for  consideration,  along  with  recommendations  concerning  curriculum 
breadth,  experience  waivers  and  future  study. 

Meeting  Research  Objectives 

The  first  research  objective  was  to  identify  and  document  the  SYS200 
course  content.  The  content  (as  defined  in  the  course  syllabus)  was  then 
compared  to  the  DoD  Program  Management  Body  of  Knowledge  (DBOK) 
proposed  by  Kobylarz.  This  comparison  found  that  the  SYS200  curricultim 
parallels  the  proposed  DBOK  in  all  but  two  areas  (Risk  Management  and 
Leadership/Personal  Skills),  which  demonstrated  support  for  the  use  of 
SYS200  as  a  cross-functional  broadening  course. 

The  second  objective  was  to  measure  the  level  of  increase  in  course 
attendees'  acquisition-related  knowledge  by  means  of  a  pretest  and  a  series 
of  three  quizzes,  with  the  average  quiz  score  serving  as  the  posttest.  The 
results  showed  that  the  overall  mean  increase  from  pretest  to  posttest  was 
substantial  (41  points,  from  41.9  to  82.9).  In  addition,  the  posttest  scores 
reflected  a  smaller  variance  than  did  the  pretest  scores,  indicating  that  the 
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range  of  student  performance  upon  course  completion  was  less  widely  spread 
than  before  attendance. 

The  third  objective  was  to  measure  the  level  of  increase  in  course 
attendees'  self-efficacy  regarding  application  of  course  knowledge  and 
techniques  in  their  work  setting,  which  was  accomplished  by  means  of  a  pre¬ 
course  survey  and  a  post-course  survey.  The  results  of  this  measure  also 
showed  substantial  increases  from  pretest  to  posttest  (overall,  fmm  40.5  to  70 
percent),  as  well  as  less  variability  in  the  post-course  survey  scores. 

The  fourth  objective  was  to  measure  supervisors’  perceptions  of  the 
level  of  increase  in  their  subordinates'  level  of  acquisition-related  knowledge, 
efficacy,  and  performance.  This  assessment  was  conducted  by  means  of  a 
post-course  survey  mailed  to  supervisors  three  months  after  their 
subordinates  completed  SYS200.  The  overall  results  of  this  measure 
indicated  moderate  improvement  in  all  three  categories,  which  also  served  to 
reinforce  the  findings  of  the  other  two  measures. 

The  fifth  objective  was  to  identify  differences  in  the  results  of  the  three 
measures  which  could  be  attributed  to  differences  by  functional  discipline,  as 
stated  in  the  research  hypotheses.  These  hypotheses  were  tested  primarily 
by  means  of  nonparametric  single-factor  ANOVAs.  The  results  showed  that, 
within  the  levels  of  confidence  indicated  by  the  reported  p-values,  none  of  the 
differences  in  scores  were  attributable  to  functional  discipline;  the  single 
exception  to  this  finding  was  in  the  knowledge  posttest,  in  which  members  of 
the  Acquisition  Logistics  group  whose  duties  were  sustainment-oriented 
scored  significantly  lower.  Despite  this  difference,  the  amount  of  increase  in 
acquisition-related  knowledge  for  all  functional  disciplines  {including 
Acquisition  Logistics)  was  essentially  equivalent. 


87 


There  were  also  some  differences  in  the  self-efficacy  survey  scores  by 
functional  discipline,  but  in  this  measure,  the  differences  were  fotmd  to  be 
insignificant.  The  amount  of  increase  in  self-efficacy  for  all  fiuictional 
disciplines  was  again  essentially  equivalent. 

The  results  of  the  three  measures  compared  favorably  with  each  other, 
both  qualitatively  (in  terms  of  similarity  of  findings)  and  quantitatively  (by 
means  of  correlation  of  measures).  The  consistency  of  results  demonstrated 
good  support,  not  only  for  the  findings  or  the  measures  themselves,  but  also 
for  the  research  construct. 

The  final  objective  was  to  identify  other  evidence  of  SYS200 
broadening  effectiveness.  The  study  found  evidence  that  functional 
discipline  groups  did  better  on  subject  matter  specific  to  their  discipline,  but 
that  no  particular  functional  discipline  did  significantly  better  at  the 
aggregate  level.  In  addition,  there  were  overall  increases  in  awareness  of  the 
acquisition  roles  and  responsibilities  of  all  functional  disciplines. 

General  Conclusions 

The  goal  of  this  study  was  to  determine  if  SYS200  was  effective  as  a 
broadening  course  for  all  functional  disciplines.  The  construct  for  evaluating 
the  course's  effectiveness  by  means  of  three  complementary  measures  was 
based  on  the  premise  that  the  SYS200  curriculum  was  sufficiently  broad,  as 
shown  in  the  comparison  to  the  DoD  Program  Management  Body  of 
Knowledge  (DBOK). 

The  results  of  the  research  demonstrated  that  SYS200  is  performing 
well  as  an  overview  of  the  acquisition  process  for  all  functional  disciplines. 
Furthermore,  equivalent  group  increases  in  knowledge  and  self-efficacy,  as 
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well  as  overall  increases  in  awareness  of  all  functional  disciplines' 
acquisition  roles  and  responsibilities,  suggest  that  SYS200  is  functioning 
effectively  as  a  broadening  course. 

CmricHlmn  Bieadth 

While  the  study  results  indicated  that,  in  general,  the  course  is 
achieving  its  intended  objectives,  the  research  also  found  that  some  aspects 
of  acquisition  management  (specifically.  Risk  Management  and  Leadership/ 
Personal  Skills)  were  not  addressed  in  detail  in  the  SYS200  course  content. 

In  addition,  the  low  scores  posted  by  the  Acquisition  Logistics  group  suggest 
that  the  lack  of  subject  material  relative  to  sustainment-oriented  activities 
may  be  negatively  affecting  the  performance  of  some  members  of  that 
functional  discipline.  In  general,  though,  the  course  content  is  representative 
of  the  overall  acquisition  process  as  currently  defined. 

Significance  of  Experience 

Acquisition  experience  was  foimd  to  be  a  negligible  factor  in 
determining  knowledge  and  self-efficacy  measure  scores.  On  the  knowledge 
pretest,  this  was  especially  true  for  attendees  who  had  at  least  four  years  of 
experience,  for  whom  each  additional  year  of  experience  would  only  accoimt 
for  an  increase  of  two-tenths  of  a  point.  On  the  posttests,  experience  became 
even  less  of  a  factor.  These  results  suggest  that  having  several  years  of 
acquisition-related  experience,  while  somewhat  helpful,  does  not  necessarily 
ensure  a  broad  base  of  acquisition  knowledge  or  capabilities. 
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Recommendations 


Curriciilum  Breadth 

As  described  in  Chapter  I,  the  field  of  acquisition  management  is  both 
diverse  and  dynamic,  and  is  certainly  one  of  the  most  challenging  areas  in 
management  today.  In  light  of  these  challenges  and  study  findings,  the 
SYS200  faculty  should  consider  subject  material  relative  to  Risk 
Management,  Leadership/Personal  Skills,  and  sustainment-oriented 
Logistics  for  incorporation  into  the  course  curriculum. 

The  comparison  of  the  SYS200  curriculum  with  the  proposed  DBOK 
suggests  that  Risk  Management  and  Leadership/Personal  Skills  should  be 
part  of  a  comprehensive  overview  of  acquisition  management. 

Study  findings  of  significant  differences  attributed  to  the  sustainment- 
oriented  portion  of  the  Acquisition  Logistics  group  suggest  that  some  recent 
Air  Force  developments  are  not  yet  reflected  in  the  course  content.  The 
merger  of  Air  Force  Logistics  Command  and  Air  Force  Systems  Command 
into  Air  Force  Materiel  Command  (AFMC)  brings  acquisition  and 
sustainment  activities  closer  together  --  as  does  implementation  of  AFMC's 
Integrated  Weapons  System  Management  (IWSM)  concept,  in  which  the 
iiltimate  goal  is  a  single  organization  responsible  for  the  "cradle-to-grave" 
acquisition  and  support  of  weapon  systems.  In  light  of  these  developments, 
SYS200  should  be  updated  to  reflect  this  philosophy. 

Experience  Waivers 

Acquisition  experience  was  shown  not  to  be  a  significant  factor  in 
study  measures.  In  light  of  this  finding,  blanket  experience  waivers  for  the 
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APDP  certification  requirement  for  completion  of  SYS200  should  not  be 
granted.  The  results  of  this  study  which  argue  against  granting  blanket 
experience  waivers  are  appropriately  reflected  by  the  following  comment 
from  the  supervisor  survey: 

This  student  has  14  years  experience  with  7  of  those  in  a 
[system  program  oflBce].  He  has  a  master's  from  AFIT  in 
Logistics,  took  this  course  to  fill  the  square,  and  yet  still  got 
something  out  of  the  course. 

Alternatively,  some  other  measure  (besides  years  of  experience)  which 
could  assess  the  breadth  of  an  individual's  acquisition-related  knowledge  and 
capabilities  would  be  appropriate  for  purposes  of  waiving  certification 
requirements. 

Further  Study 

Continuation  of  this  study  has  two  potential  advantages:  1)  additional 
observations  would  lead  to  greater  statistical  power  and  would  allow 
multivariate  analysis  as  a  more  complete  matrix  of  functional  discipline 
versus  years  of  experience  becomes  available;  and  2)  additional  changes 
made  to  the  course  could  be  compared  to  the  "baseline"  established  in  this 
study  as  a  means  of  assessing  improvement  of  the  educational  process. 
Expanding  this  study  to  satellite  locations  would  further  add  to  sample  size 
and  would  allow  for  consideration  of  potential  differences  in  quality  by 
location. 

Adapting  a  similar  methodology  for  evaluating  other  courses 
mandated  by  the  APDP  program  could  provide  an  assessment  of  the 
effectiveness  of  each  course  relative  to  its  APDP  objective  and  a 
comprehensive  measure  of  the  effectiveness  of  the  APDP  program. 
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Each  functional  discipline  has  a  different  number  of  people  who  will 
ultimately  require  SYS200  for  APDP  certification.  Functional  discipline 
representation  at  the  AFIT  in-residence  course  was  shown  to  be  generally 
consistent  with  the  overall  requirements.  The  two  exceptions  found  were  the 
Financial  Management  and  Contracting  functional  disciplines.  The 
Contracting  discrepancy  is  easily  explained,  as  SYS200  is  "desired"  but  not 
"required"  for  certification  in  that  functional  discipline. 

The  Financial  Management  functional  discipline  was  not 
proportionately  represented,  perhaps  because  its  members  do  not  require 
SYS200  completion  until  the  highest  APDP  certification  level.  However, 
since  this  is  also  true  for  many  other  functional  disciplines,  it  is  not  clear 
why  fewer  persons  from  Financial  Management  are  attending.  (It  is 
noteworthy  that  the  single  supervisor  comment  on  course  availability  was 
from  a  supervisor  whose  subordinate  is  in  Financial  Management.) 

Further  investigation  should  be  conducted  to  assess  how  well  SYS200 
will  be  able  to  meet  its  future  requirements  for  attendance. 


Timing  of  SYS20Q  Attendance 

This  study  found  that  41  percent  of  the  sample  had  eight  or  more  years 
of  acquisition  experience  prior  to  attending  the  course.  Although  this  may  be 
a  function  of  "catching  up"  to  APDP  requirements,  interim  guidance  from 
SAF/AQ  now  calls  for  most  functional  disciplines  to  complete  SYS200  for 
Level  III  certification  (which  would  typically  come  later  in  a  person's  career). 
For  this  reason,  experience  levels  of  attendees  may  continue  to  be  high. 


Further  study  should  investigate  an  "optimal"  time  for  attendance  in 
light  of  APDP  goals,  and  consider  whether  SYS200's  overview  of  the 
acquisition  process  should  be  communicated  to  acquisition  personnel  earlier 
in  their  careers. 

Correlation  Of  Measures 

While  correlating  the  results  of  the  three  measures  wa  primarily 
intended  to  strengthen  the  research  and  construct  validity,  this  process  also 
generated  another  finding  of  interest. 

The  single  case  in  which  knowledge  and  self-efficacy  scores  were  not 
closely  correlated  occurred  in  the  Communications/Computers  functional 
discipline,  where  persons  tended  to  score  higher  on  the  knowledge  tests  but 
lower  on  the  self-efficacy  measure  (though  the  differences  from  the  other 
groups  were  not  statistically  significant). 

One  possible  explanation  for  this  finding  is  that  communications  and 
computer  systems  which  are  not  "embedded"  in  weapons  systems  are  often 
procured  through  a  different  acquisition  process  governed  by  the  700-series 
Air  Force  regulations.  Lack  of  familiarity  with  the  DoD  5000-series  process 
taught  in  SYS200  may  be  affecting  self-assessment  of  the  ability  to  perform 
these  tasks,  despite  demonstration  of  sufficient  learning  on  the  knowledge 
measure. 

Further  study  should  investigate  possible  causes  of  this  phenomenon, 
assess  the  differences  between  these  two  acquisition  processes  ^nd  (if 
necessary)  reflect  upon  APDP  certification  requirements  in  the  face  of  any 
discovered  differences. 
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Summary 


This  study  foimd  that  SYS200  is  performing  effectively  as  a 
broadening  course,  and  is  well-suited  for  its  role  as  a  required  course  for 
certification  in  all  acquisition  functional  disciplines.  This  study  also 
identified  areas  for  incorporation  into  the  SYS200  curriculum  which  could 
enhance  its  performance  relative  to  APDP  goals.  Finally,  acquisition 
experience  was  foimd  not  to  be  a  significant  factor  in  determining  breadth  of 
knowledge  and  capabilities. 

The  results  of  this  study  support  the  concept  of  acquisition 
professional  development.  In  the  current  environment  of  reductions  in  force 
structure  and  budget  authority,  it  is  more  important  than  ever  to  continue 
the  focus  on  developing  and  maintaining  a  highly  educated  and  motivated 
acquisition  workforce. 
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Appendix  A:  SYS2Q0  Course  Syllabus 


Section  1 
COURSE  OBJECTIVES: 

The  purpose  of  AFIT  Course  SYS  200,  Acquisition  Planning  and 
Analysis  is  to  prepare  program/project  officers  for  basic 
management  level  task  accomplishment  in  a  program  office. 

The  objectives  of  the  course  are  for  each  student  to: 

a.  Understand  the  individual  management  processes  involved 
in  weapon  systems  acquisition. 

b.  Understand  the  concept  of  an  integrated  team  approach 
to  acquisition. 

c.  Develop/refine  skills  in  planning,  analyzing,  executing, 
and  controlling  acquisition  projects. 


Section  2 

COURSE  DESCRIPTION: 

SYS  200  consists  of  lectures,  discussions,  individual  and 
group  exercises  in  three  main  subject  areas: 

I.  Planning  the  Program 

Includes  an  overview  of  the  presentations  on  the 
acquisition  process,  acquisition  contracting,  financial  planning, 
work  breakdown  structure  generation  and  network  analysis. 

II.  Achieving  Program  Objectives 

Includes  the  interfaces  with  functional  groups  (the 
Integrated  Team  Approach)  and  introduces  these  various  groups  and 
their  unique  disciplines:  systems  engineering,  configuration 
management,  software  management,  test  and  evaluation,  acquisition 
logistics  and  contracting.  In  addition,  a  group  exercise  called 
the  Integrative  Exercise  reinforces  these  ideas  through  student 
group  interaction  to  plan  out  a  given  project  (Block  I)  and  task 
various  groups  of  functional  specialists  (Block  II)  to  get  the 
project  done. 

III.  Controlling  the  Program 

Includes  lectures  and  exercises  to  introduce  various 
control  mechanisms  and  practice  these  disciplines:  program 
control,  program  review  (briefing  of  program  status  to  a  new 
program  director) ,  cost  estimating  and  performance  measurement 
(cost/schedule  control  system  criteria)  analysis  techniques. 
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Section  3 

COURSE  OUTLINE 

Number 

Lesson  or  Activity  of  Hrs 

a.  Course  Registration/Orientation  .  1.5 

b.  The  Acquisition  Process  .  5.5 

c.  Program  Management  .  1.5 

d.  Acc[uisition  Program  Planning  . . .  3 

e.  Acquisition  Program  Planning  Exercise  .  3 

f.  Financial  Management  .  3 

g.  Financial  Management  Exercise  (POM  Ex) .  3 

h.  Work  Breakdown  Structures  .  1 

i.  WBS  Exercise  .  2 

j.  Scheduling  .  1.5 

k.  Networking  Exercise  .  2.5 

l.  Exam  #1  and  Review  .  1.5 

m.  Integrative  Planning  Exercise  . . .  12 

n.  Systems  Engineering  .  4 

o.  Configuration  Management  .  2 

p.  Software  Management  .  2 

q.  Test  &  Evaluation  .  3 

r.  Acquisition  Logistics  .  4 

s.  Technical  Package  Requirements  Formulation  .  3 

t.  SOW/CDRL  Exercise  .  3 

u.  Contracting  .  2 

V.  Dealing  with  Contractors  .  2 

w.  Facilities  Acquisition  Management  .  1.5 

X.  Manufacturing  Management  . 2 
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y.  Exam  #2  and  Review  .  1.5 

z.  Program  Control  .  1 

aa.  Cost  Estimating  .  2.5 

bb.  Cost  Estimating  Exercise  .  1.5 

cc.  Performance  Measurement  (Including  exercise)  .  7.0 

dd.  Final  Exam  and  Review  .  1.5 

ee.  Critique  of  Program  Review  and  Integrative  Exercise  ..  1 

ff.  Course  Critique,  Summary,  and  Closing  Comments  .  1 


Total  ...  90 
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Section  4 


ADMINISTRATIVE  INFORMATION 
1)  ATTENDANCE  POLICY; 

While  attending  SYS  200,  all  students  are  under  the 
administration  of  the  Dean,  School  of  Systems  and  Logistics.  The 
Dean  has  prescribed  that  students: 

a.  Will  report  on  time  for  and  attend  all  course 
presentations  and  exercises. 

b.  Will  be  excused  from  attendance  only  for  emergency  or 
health  reasons  (student  or  immediate  family  member(s)). 

c.  Will  not  be  excused  from  any  class  session  to  return  to 
their  job/home  duty  station,  to  attend  any  courses  at  local 
civilian  colleges,  to  attend  meetings  or  briefings  or  normal 
duty,  or  to  conduct  personal  business  (medical/dental 
appointments,  vehicle  maintenance,  committee  meetings,  sports, 
etc. )  . 


d.  Will  notify  the  Course  Director  or  Class  Leader  of 
location  or  destination  if  leaving  the  local  area  during 
weekends . 


e.  Must  obtain  prior  Course  Director  or  Class  Leader 
permission  to  be  absent  from  any  class  session. 

2)  INSTRUCTIONAL  METHODS.  ACADEMIC  FREEDOM.  AND  NONATTRIBUTIQN 

policy; 

Lectures  are  presented  by  faculty  members  of  the  School  of 
Systems  and  Logistics  and  by  selected  guest  speakers. 
Instructional  methods  used  include  lecture,  discussion,  in-class 
exercises  and  homework  exercises.  Homework  is  in  the  form  of 
readings  and  take-home  exercises.  The  Integrative  Exercise  is  a 
group  project  done  partly  in  class.  Additional  time  outside  of 
class  is  required  to  complete  the  project. 

Academic  Freedom  is  defined  as  the  privilege  of  debate  with 
discretion  on  any  subject  related  to  curricula  within  the  school 
forum. 

Nonattribution  means  treating  statements  made  in  the  school  forum 
as  privileged  information  not  to  be  attributed  to  a  specific 
individual.  (That  is,  outside  of  the  school  forum,  we  do  not 
quote  a  lecturer's  or  fellow  student's  remarks:  Such  as,  "Joe 
Smith  said  in  class,  'program  X  did  not  comply  with  AFRXX-X. '") . 
Within  these  guidelines,  each  student  is  encouraged  to 
participate  and  freely  discuss  the  subject  matter  presented. 
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3)  STUDENT  PREPARATION;  Students  are  expected  to: 


a.  Read  scheduled  assignments  in  advance. 

b.  Remain  alert  and  attentive  in  class. 

c.  Take  notes.  Notes  in  your  own  words  add  greatly  to  your 
retention  of  material  presented. 

d.  Participate  actively  in  discussions,  but  please  do  not 
monopolize  or  side  track  them. 

e.  Remember  that  some  of  the  speakers  are  our  guests  and 
are  contributing  their  time  and  efforts  to  aid  our 
program.  Treat  them  accordingly. 

4)  CLASS  LEADER  RESPONSIBILITIES: 

The  senior  military  student  will  be  designated  the  course 
Class  Leader.  The  Class  Leader's  responsibilities  include: 

a.  Representing  the  Course  Director. 

b.  Monitoring  class  attendance. 

c.  Monitoring  class  care  of  the  classroom  (housekeeping) . 

d.  Organizing  any  social  functions  desired  by  the  class. 

e.  Serving  as  a  focal  point  to  provide  student  feedback  to 
the  Course  Director  and  the  Head  of  the  Department  of  System 
Acquisition  Management. 


5)  STUDENT  EVALUATION  OF  THE  COURSE; 

Four  principal  means  of  obtaining  student  evaluations  of 
the  course  content  and  conduct  are  provided  by: 

a.  Direct  feedback  to  the  Course  Director/Assistant  Course 
Director  at  any  time  and/or  at  the  final  class  session. 

b.  Student  end-of-course  written  appraisals  submitted  at 
the  final  class  session. 

c.  Student  Class  Leader  debriefing  (optional)  on  the  last 
class  day  with  the  Head  of  the  Department  of  System  Acquisition 
Management . 

d.  Post-course  (approximately  six  months)  surveys  of selected 
classes. 
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s?gti<?n  5 

EVALUATION  CRITERIA/GRADING  POLICY 

SYS  200  grading  is  on  a  satisfactory/unsatisfactory 
(pass/fail)  basis.  The  minimum  acceptable  (satisfactory) 
averages  are  a  70%  overall  average  for  the  entire  course  with  a 
average  of  60%  on  all  quizes.  Failure  will  occur  should  either 
of  these  minimums  not  be  reached.  There  is  no  provision  to 
retake  a  failed  final  exam.  The  only  option  is  to  retake  the 
entire  course  by  special  permission  from  the  Course  Director. 

a.  The  "I"  grade  will  be  given  for  failure  to  complete  the 
required  course  work.  The  "I"  recipient  has  six  (6)  weeks  after 
the  original  course  completion  data  to  complete  such.  Two 
options  are  inherent  with  the  award  of  an  "I"  grade. 

(1)  Option  A:  If  you  submit  unfinished  work  for 
grading,  and  it  is  satisfactory,  the  "I"  grade  will  be  changed  to 
Satisfactory  (S) .  If  submitted  and  the  work  is  unsatisfactory, 
you  will  receive  the  grade  Unsatisfactory  (U) . 

(2)  Option  B:  If  you  do  not  submit  uncompleted  course 
work  and  desire  for  valid  reasons  not  to  complete  it  or  if  you 
cannot  realistically  complete  the  work  within  a  six  week  time 
frame,  you  will  receive  a  "W”  grade. 

b.  The  "W”  grade  will  be  given  to  those  who  must  withdraw 
from  a  course  prior  to  completing  recjuirements.  This  grade  is 
not  used  as  a  substitute  for  a  failing  grade  but  is  only  used 
when  attendance  is  shortened  by  illness,  recall,  or  other  valid 
reasons. 

c.  Students  will  be  evaluated  in  the  three  areas  listed 
below: 


EVALUATION  AREA  %  OF  GRADE 

Quiz  #1  20 
Quiz  #2  20 
Quiz  #3  20 
Integrative  Exercise  15 
Other  Exercises  25 


NOTE:  (Course  overall  minimum  passing  average  70%) 

(Minimum  passing  score  for  the  quizes:  60%  Average) 
(Must  meet  both  of  these  criteria  to  pass  the  course) 


100 


Section  6 
COURSE  SCHEDULE 


The  detailed  class  schedule  for  the  course  varies  with  each 
offering.  Therefore,  each  student  is  given  a  copy  of  the  class 
schedule  during  the  class  registration  period. 

Section  7 

INDIVIDUAL  LESSON  PLANS 

The  remainder  of  this  document  is  a  detailed  description  of 
each  lesson  in  the  SYS  200  course.  The  following  is  a  simple 
explanation  of  key  parts  of  the  syllabus  lesson  plan  page  that 
should  be  of  greatest  interest  to  the  students. 

a.  Time:  Approximate  number  of  classroom  hours  to  be  spent 
on  the  lesson. 

b.  Lesson  Objective:  A  broad  statement  of  the  ultimate 
student  outcome. 

c.  References:  Major  documents  used  as  source  data  for 
lesson  content. 

d.  Instructional  Material:  A  listing  of  the  materials  the 
student  is  provided  as  a  routine  part  of  the  curriculum. 

However,  student  may  receive  additional  supplemental  materials 
during  the  conduct  of  the  class.  Unless  otherwise  specified, 
this  supplemental  material  is  optional  and  nontestable. 

e.  Student  Preparation:  Actions  the  student  must 
accomplish  before  a  particular  lesson  is  formally  presented  in 
class. 

Section  8 

CHEATING 

If  a  student  is  suspected  of  cheating  on  an  assignment  or 
examination,  he  or  she  may  receive  a  failing  grade  for  (1)  the 
assignment,  (2)  the  examination,  or  (3)  the  course.  If  the 
student  feels  the  grade  is  not  justified,  he  or  she  may  appeal 
first  to  the  course  director  for  reconsideration.  Secondly,  an 
appeal  may  be  made  to  the  department  head  who  will  review  the 
circumstances  and  if  a  conflict  still  exists,  will  request  a 
faculty  board  be  convened  to  investigate  the  charge  and  render  a 
recommendation.  Upon  receipt  of  the  board's  recommendation,  the 
department  will  make  the  final  decision. 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

LESSON  TITLE:  The  Acquisition  Process  (I-l) 

TIME:  5.5  hrs 

MODULE  OBJECTIVE (S) :  The  Module  Objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  Recjuirements  Evolution 
Process . 

2.  The  student  will  understand  the  process  of  program  phase 
transition  (milestones) . 

3.  The  student  will  understand  the  objective (s)  of  the  various 
phases  of  the  acquisition  life  cycle. 

4.  The  student  will  distinguish  among  the  four  categories  of 
acquisition  programs. 

5.  The  student  will  know  the  structure  of  the  Air  Force 
Acquisition  Executive  System. 

6.  The  student  will  understand  the  acquisition  related 
responsibilities  of  the  following  players  /  boards  / 
organizations:  Defense  Acquisition  Executive  (DAE) ,  Air  Force 
Acquisition  Executive  (AFAE) ,  Program  Executive  Officer  (PEO) , 
Program  Director/Manager  (PD/PM) ,  Commanders  of  the  Acquisition 
Commands,  Defense  Acquisition  Board  (DAB),  Air  Force  Systems 
Acquisition  Review  Council  (AFSARC) ,  and  Joint  Requirements 
Oversight  Council  (JROC) . 

7.  The  student  will  understand  the  purpose  and  basic  content  of 
the  following  acquisition  program  documents  /  requirements: 
Mission  Need  Statement  (MNS) ,  System  Requirements  Document  (SRD) , 
Operational  Requirements  Document  (ORD) ,  Acquisition  Program 
Baseline  (APB),  Exit  Criteria,  Acquisition  Information  System 
(AIS) ,  Program  Management  Directive  (PMD) ,  Integrated  Program 
Summary  (IPS),  Integrated  Program  Assessment  (IPA),  Acquisition 
Decision  Memorandum  (ADM) . 

8.  The  student  will  understand  the  philosophy  underlying  the 
interface  of  the  Requirements  Generation  System,  the  Acquisition 
Management  System,  and  the  Planning,  Programming,  and  Budgeting 
System  (DoD's  three  major  decisionmaking  support  systems). 

9.  The  student  will  understand  the  process  of  transitioning  a 
program  to  Operations  and  Deployment. 
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10.  The  student  will  identify  the  Department  of  Defense,  and  Air 
Force  Directives  and  Regulations  on  systems  acquisition. 

11.  The  student  will  know  the  DoD  guidance  (MIL-HDB-248B)  on  the 
concept  of  acquisition  streamlining. 

12.  The  student  will  understand  the  overall  modification 
management  process  of  an  acquired  system  throughout  its  life 
cycle. 

13.  The  student  will  know  the  major  modification  management 
responsibilities  of  the  Systems  Program  office  /  Air  Logistics 
Center  /  and  user. 


METHOD  OF  iNSTRUCTiOMi  Lecture/Discussion 
INSTRUCTIONAL  MATERIALS;  Lesson  viewgraphs 

AUDIO-VISUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES !  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

LF.ssoM  TTTT.R;  Program  Management  (1-2) 

TIME:  1.5  hrs 

MODULE  OBJECTIVE (S) :  The  Module  Objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  identify  the  role  definition  of  the  Project 
Officer  in  systems  acquisition. 

2.  The  student  will  distinguish  between  management  and 
acc[uisition  management. 

3.  The  student  will  identify  the  dominant  and  subordinate 
objectives  of  a  Systems  Program  Office  (SPO) . 

4.  The  student  will  distinguish  between  a  typical  single  Systems 
Program  Office  (SPO)  and  a  typical  multiple  (basket)  Systems 
Program  Office  (SPO) . 

5.  The  student  will  understand  the  interface  between  a  Systems 
Program  Office  and  other  acquisition  related  organizations. 

6.  The  student  will  Identify  the  key  factors  that  determine  the 
success  of  a  program. 

7.  The  student  will  understand  the  nature  of  a  Project  Officer's 
responsibilities  (tasks) . 

8.  The  student  will  identify  the  skills/attributes  that  are 
essential  for  a  Project  Officer. 

9.  The  student  will  identify  the  keys  to  success  for  a  Project 
Officer. 

10.  The  student  will  understand  the  general  methods  available  to 
assist  the  Project  Officer  in  problem  solving  /  decision  making. 
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METHOD  OF  INSTRUCTION:  Lecture/Discussion 
INSTRUCTIONAL  MATERIALS;  Lesson  viewgraphs 

AUDIO-VISUAL  AIDS!  Transparency,  overhead  projector,  screen,  and 

chalkboard 

INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES ;  Reading 

REQUIRED  STUDENT  REPARATION;  Lesson  Readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

LESSON  TITLE;  Acquisition  Program  Planning  (1-3) 

TIME;  6  hrs 

MODULE  OBJECTIVE(S) ;  The  Module  Objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  top  level  processes 
(activities  /  decisions  )  required  prior  to  award  of  an 
acquisition  contract. 

2 .  The  student  will  understand  the  concept  of  accpiisition 
strategy . 

3.  The  student  will  recognize  the  purpose  and  content  of  the 
acquisition  plan. 

4.  The  student  will  understand  the  activities  required  for 
development  and  release  of  a  Request  for  Proposal  (RFP) . 

5.  The  student  will  understand  the  activities  of  Source 
Selection. 

6.  The  student  will  identify  contracting  decisions  that  must  be 
made  during  acqpiisition  planning. 

7.  The  student  will  understand  the  acquisition  streamlining 
concepts  applied  to  acquisition  planning  (RFP  and  Source 
Selection) . 

8.  The  student  will  develop  an  Acquisition  Strategy  and 
Acquisition  Plan  for  a  simulated  program. 


METHOD  OF  INSTRUCTION:  Lecture/Discussion 
INSTRUCTIONAL  MATERIALS;  Lesson  viewgraphs 

AUDIO-VISUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES :  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  Readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 


SYS  200 

LESSON  TITLE;  Financial  Management  (1-4) 
TIME;  6  hrs 


MODULE  OBJECTIVE (S) :  The  Module  Objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  Six  Year  Defense  Program 
(SYDP) . 

2.  The  student  will  understand  the  Biennial  Planning, 

Programming,  and  Budgeting  System  (BPPBS) . 

3.  The  student  will  understand  the  Enactment  Phase  of  the  Federal 
Budget  Process. 

4.  The  student  will  understand  the  Execution  Phase  of  the  Federal 
Budget  Process. 

5.  The  student  will  understand  the  role  of  the  Project  Officer  in 
programming  resources. 

6.  The  student  will  understand  the  concept  of  the  Program 
Objective  Memorandum  (POM) . 

7.  The  student  will  forecast  financial  requirements  by  preparing 
a  typical  System  Program  Office  (SPO)  Program  Decision  Package 
(PDP) ,  Program  Objective  Memorandum  (POM) . 

METHOD  OF  INSTRUCTION!  Lecture/Discussion/Group  Exercise 

INSTRUCTIONAL  MATERIALS!  Lesson  viewgraphs  and  Exercise 

AUDIO-VIDUAL  AIDS!  Overhead  projector,  screen,  and  chalkboard 

INSTRUCTIONAL  EQUIPMENT:  Not  applicable 

REFERENCES!  Readings 

REQUIRED  STUDENT  PREPARATION!  Lesson  readings 

Read  POM  Exercise 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

LESSON  TITLE:  Work  Breakdown  Structure  (1-5) 

TIME;  3  hrs 

MODULE  OBJECTIVES  (S) :  The  module  objective (s)  for  each  student 
are  as  follows: 

1.  The  student  will  understand  the  concept  of  the  Work  Breakdown 
Structure  (WBS)  in  acquisition  planning. 

2.  The  student  will  understand  the  WBS  structure. 

3.  The  student  will  understand  how  the  Cost  Account/Work  Package 
structure  fits  into  the  WBS. 

4.  The  student  will  understand  how  the  different  types  of  Work 
Breakdown  Structures  are  used  in  acquisition  program  management. 

5.  The  student  will  understand  the  relationship  between  the  WBS 
and  the  Statement  of  Work  (SOW) . 

6.  The  student  will  understand  the  relationship  between  the  WBS 
and  Contractor  Performance  Measurement  (C/SCSC/Cost  Performance 
Report) . 

7.  The  student  will  understand  the  benefits  of  using  Work 
Breakdown  Structures . in  acquisition  program  management. 

8.  The  student  will  understand  the  importance  of  the  Work 
Breakdown  Structure  in  providing  a  foundation (thread)  for 
acquisition  program  management. 

9.  The  student  will  develop  a  Summary  Work  Breakdown  Structure 
(WBS)  for  a  simulated  acquisition  program. 

10.  The  student  will  develop  a  Project  Summary  Work  Breakdown 
Structure  (WBS)  for  a  simulated  acquisition  program 
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METHOD  OF  INSTRUCTION!  Lecture/Discussion/Group  Exercise 
INSTRUCTIONAL  MATERIALS;  Lesson  viewgraphs  and  Mil-S-881A 
AUDIO-VIDUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES ;  Readings 

REQUIRED  STUDENT  PREPARATION:  Lesson  Readings 

Read  WBS  Exercise 
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ACQUISITION  PLANNING  AND  ANALYSIS 


SYS  200 

LESSON  TITLE;  Scheduling  (1-6) 

TIME:  4  hrs 

MODULE  OBJECTIVE (S) :  The  module  objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  distinguish  between  the  different  forms  of 
scheduling  techniques  (Gantt  Chart,  Milestone  Chart,  Flow  Process 
Chart,  Line  of  Balance,  and  Network  Analysis) . 

2.  The  student  will  understand  the  concept  of  using  Integrated 
Program  Master  Schedules  within  the  program  office. 

3.  The  student  will  identify  the  strengths  and  weaknesses  of  the 
different  forms  of  scheduling  techniques. 

4.  The  student  will  understand  the  process  of  acquisition  project 
networking. 

5.  The  student  will  identify  the  four  basic  steps  used  to 
construct  a  network  schedule. 

6.  The  student  will  understand  the  concept  of  the  critical  path. 

7.  The  student  will  understand  the  concept  of  slack  time. 

8.  The  student  will  know  the  software  tools  available  to  assist 
the  Project  Officer  in  scheduling. 

9.  The  student  will  apply  the  technique  of  network  analysis  to  a 
simulated  acquisition  program. 

10.  The  student  will  understand  the  different  solutions 
availiable  to  correct  a  scheduling  problem  discovered  in  the 
network  analysis  process. 
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METHOD  OF  INSTRUCTION;  Lecture/Discussion/Group  exercise 
INSTRUCTIONAL  MATERIALS;  Lesson  viewgraphs 

AUDIO-VISUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 

INSTRUCTIONAL  EQUIPMENT;  Not  applicable 

REFERENCES ;  Readings/Exercise 

REQUIRED  STUDENT  PREPARATION;  Lesson  readings 

Read  Case  Exercise 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

LESSON  TTTT.R;  Acquisition  Program  Planning  (1-7) 

TIME;  12  hrs 

MODULE  OBJECTIVE(S) :  The  Module  Objective(s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  top  level  processes 
(activities  /  decisions  )  required  prior  to  award  of  an 
acquisition  contract. 

2.  The  student  will  understand  the  concept  of  acquisition 
strategy . 

3.  The  student  will  recognize  the  purpose  and  content  of  the 
acquisition  plan. 

4.  The  student  will  understand  the  activities  required  for 
development  and  release  of  a  Request  for  Proposal  (RFP) . 

5.  The  student  will  understand  the  activities  of  Source 
Selection. 

6.  The  student  will  identify  contracting  decisions  that  must  be 
made  during  acquisition  planning. 

7.  The  student  will  understand  the  acquisition  streamlining 
concepts  applied  to  acquisition  planning  (RFP  and  Source 
Selection) . 

8.  The  student  will  develop  an  Acquisition  Strategy  and 
Acquisition  Plan  for  a  simulated  program 

METHOD  OF  INSTRUCTION;  Lecture/Discussion/Group  exercise 

INSTRUCTIONAL  MATERIALS;  Exercise  package.  Mil  Standards,  and 

Mil-HDBK-245B 

AUDIO-VISUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Computer 
REFERENCE ;  Exercise 

REQUIRED  STUDENT  PREPARATION;  Read  Exercise  before  class. 

Submit  a  package  of  all  taskings  required  by  the  course  director. 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

LESSON  TTTT.R!  Systems  Engineering  (II-l) 

TIME;  3  hrs 

MODULE  OBJECTIVE (S) :  The  module  objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  basic  concept/philosophy  of 
Systems  Engineering. 

2.  The  student  will  understand  the  content  of  the  Systems 
Engineering  Management  Plan  (SEMP) . 

3.  The  student  will  understand  the  Systems  Engineering  Process. 

4.  The  student  will  differentiate  between  the  Functional, 
Allocated,  and  Product  Baselines. 

5.  The  student  will  understand  the  purpose  of  each  of  the  major 
reviews  /  audits  in  the  Systems  Engineering  Process. 

6.  The  student  will  understand  the  documentation  associated  with 
each  major  design  review  /  audit. 

7.  The  student  will  know  the  sequential  time  phasing  of  the  major 
reviews  /  audits. 

8.  The  student  will  understand  the  importance  of  risk  management 
in  the  Systems  Engineering  process. 

9.  The  student  will  identify  the  major  pitfalls  of  the  Systems 
Engineering  process 
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METHOD  OF  INSTRUCTION!  Lecture/Discussion 
INSTRUCTIONAL  MATERIALS;  Lesson  viewgraphs 

AUDIO-VIDUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES ;  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  Readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

T.RSRON  TTTT.Ki  Configuration  Management  (II-2) 
TIME;  1.5  hrs 


MODULE  OBJECTIVE (S) :  The  Module  objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  what  is  meant  by  the  term 
configuration. 

2.  The  student  will  understand  the  concept  of  the  configuration 
item  (Cl) . 

3.  The  student  will  understand  the  overall  purpose  of 
configuration  management  in  weapon  system  acquisition. 

4.  The  student  will  understand  the  functions  of  configuration 
management  in  weapon  system  acquisition. 

5.  The  student  will  understand  the  overall  concept  of  progressive 
design  control . 

6.  The  student  will  understand  the  concept  of  configuration 
management  baselines  (functional,  allocated,  and  product). 

7.  The  student  will  distinguish  between  a  Type  A  (Systems) 
specification,  Type  B  (Development)  specification,  Type  C 
(Product)  specification.  Type  D  (Process)  specification,  and  Type 
E  (Material)  specification. 

8.  The  student  will  understand  the  process  required  to  change  a 
specification. 

9.  The  student  will  understand  the  use  of  the  different  types  of 
drawings. 

10.  The  student  will  understand  the  purpose  of  design  reviews/ 
configuration  audits. 

11.  The  student  will  understand  the  difference  between  an 
Engineering  Change  Proposal  (ECP)  and  an  Advanced  Change  Study 
Notice  (ACSN) . 
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12.  The  student  will  know  the  content  of  a  typical  Engineering 
Change  Proposal (ECP) . 

13.  The  student  will  understand  the  function  of  the 
Configuration  Control  Board  (CCB) . 


14.  The  student  will  understand  the  concept  of  Interface 
Control . 

15.  The  student  will  understand  the  benefits  afforded  by 
using  configuration  management  in  acquisition  program  manage¬ 
ment. 

METHOD  OF  INSTRUCTION;  Lecture/Discussion 
INSTRUCTIONAL  MATERIAL;  Lesson  viewgraphs 

AUDIO-VIDUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES !  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  Materials 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

T.RSsnM  TTTT.Ri  Software  Management  (11-3) 

TIME!  2  hrs 

MODULE  OBJECTIVE (S) :  The  Module  Objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  how  Air  Force  agencies  manage 
Mission  Critical  Computer  Resources  (MCCR)  software  throughout 
the  systems  acquisition  life  cycle. 

2.  The  student  will  describe  the  eight  (8)  phases  of  the  software 
development  cycle  as  outlined  in  DoD-STD-2167A. 

3.  The  student  will  summarize  the  responsibilities  of  the 
developing,  supporting,  and  using  commands  for  Mission  Critical 
Computer  Resources  (MCCR)  software  life  cycle  management. 

4.  The  student  will  identify  the  purpose  of  a  Computer  Resources 
Working  Group  (CRWG) . 

5.  The  student  will  know  the  member  organizations  on  the  Computer 
Resources  Working  Group  (CRWG) . 

6.  The  student  will  identify  the  purpose  of  the  Computer 
Resources  Life  Cycle  Management  Plan  (CRLCMP) . 

7.  The  student  will  know  the  content  of  the  Computer  Resources 
Life  Cycle  Management  Plan  (CRLCMP) 


METHOD  OF  INSTRUCTION:  Lecture/Discussion 
INSTRUCTIONAL  MATERIALS:  Lesson  viewgraphs 

AUDIO-VIDUAL  AIDS:  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT:  Not  applicable 
REFERENCES :  Readings 

REQUIRED  STUDENT  PREPARATION ;  Lesson  Readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

T.ESSON  TITLE:  Test  and  Evaluation  (II-4) 

TIME;  2  hrs 

MODULE  OBJECTIVE (S ) :  The  Module  Objective(s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  need  for  Test  and  Evaluation 
(T&E)  in  Air  Force  weapon  system  acquisition. 

2.  The  student  will  know  the  test  responsibilities  of  the  various 
organizations  during  the  systems  acquisition  life  cycle  of  a 
system. 

3.  The  student  will  identify  the  activities  which  take  place 
during  Development  Test  and  Evaluation  (DT&E) . 

4.  The  student  will  identify  the  activities  which  take  place 
during  Initial  Operational  Test  and  Evaluation  (lOT&E) . 

5.  The  student  will  identify  the  activities  which  take  place 
during  Follow-On  Operational  Test  and  Evaluation  (FOT&E) . 

6.  The  student  will  know  the  Project  Officer's  role  during  all 
phases  (DT&E/IOT&E/FOT&E)  of  Test  &  Evaluation  during  the 
acquisition  of  a  new  system. 

7.  The  student  will  know  the  advantages  /  disadvantages  of  a 
combined  Development  Test  &  Evaluation  (DT&E)  and  Initial 
Operational  Test  &  Evaluation  (lOT&E) . 

8.  The  student  will  know  the  purpose  of  the  Test  Planning  Working 
Group  (TPWG) . 

9.  The  student  will  know  the  purpose  of  the  Test  and  Evaluation 
Master  Plan  (TEMP) . 

10.  The  student  will  know  the  purpose  of  the  Requirements 
Coorelation  Matrix  (RCM) . 

11.  The  student  will  know  the  reporting  requirements  for  the  Test 
&  Evaluation  function  of  major  systems  acquisition. 
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METHOD  OF  INSTRUCTION;  Lecture/Discussion 
INSTRUCTIONAL  MATERIALS;  Lesson  viewgraphs 

AUDIO-VIDUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES ;  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

LESSON  TITLE;  Acquisition  Logistics  (II-5) 

TIME:  2.5  hrs 

MODULE  OBJECTIVE (S) :  The  module  objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  overall  concept  of  acquisition 
logistics  planning  as  it  applies  to  the  overall  systems 
acquisition  process. 

2.  The  student  will  understand  how  acquisition  logistics  planning 
is  used  in  the  overall  acquisition  planning  process. 

3.  The  student  will  know  the  objectives  of  Integrated  Logistics 
Support  (ILS)  Program  as  they  relate  to  the  systems  acquisition 
process. 

4.  The  student  will  identify  the  ten  (10)  elements  of  logistics. 

5.  The  student  will  understand  the  Logistics  Support  Analysis 
(LSA)  process  used  to  establish  the  logistics  support  structure. 

METHOD  OF  INSTRUCTION:  Lecture/ Discuss ion 
INSTRUCTIONAL  MATERIALS;  Lesson  viewgraphs 
AUDIO-VISUAL  AIDS;  Overhead  projector,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES ;  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

T.RSRON  TTTT.F;  Technical  Package  Requirements  Formulation 

(SOW,  Data,  and  Specifications/Standards)  (II-6) 

TIME;  6  hrs 

MODULE  OBJECTIVE (S) :  The  module  objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  be  able  to  distinguish  requirements  as  to 
their  correct  placement  in  the  different  parts  of  a  contract 
(e.g.,  the  Schedule,  the  General  Provisions,  the  SOW,  the 
Contract  Data  Requirements  List  (CDRL) ,  and  the  Specification) . 

2.  The  student  will  understand  the  importance  of  the  Statement  of 
Work  (SOW)  in  the  overall  systems  acquisition  process. 

3.  The  student  will  know  the  procedures  for  writing  a  Statement 
of  Work  (SOW) . 

4.  The  student  will  know  the  Air  Force  Policy  (AFR  310-1)  on 
placing  data  tasking  and  data  preparation  instructions  in  the 
Contract  Data  Requirements  List  (CDRL) . 

5.  The  student  will  know  the  procedure  for  ordering  data  for  an 
impending  contractual  action. 

6.  The  student  will  know  the  DoD  guidance  (MIL-HDBK-245B)  on 
developing  a  Statement  of  Work  (SOW)  for  an  impending  contractual 
action. 

7.  The  student  will  distinguish  between  the  different  types  of 
Statements  of  Work. 

8.  The  student  will  understand  some  of  the  major  problems  caused 
by  a  poorly  prepared  Statement  of  Work  (SOW) . 

9.  The  student  will  understand  the  purpose  of  a  specification  / 
standard. 

10.  The  student  will  understand  the  content  of  a  specification  / 
standard. 

11.  The  student  will  distinguish  between  a  specification  and  a 
standard . 

12.  The  student  will  understand  the  requirements  of  a 
specification  for  the  development  of  a  new  system. 
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13.  The  student  will  distinguish  between  Qualification  and 
Acceptance  Test  provisions  of  specifications. 

14.  The  student  will  distinguish  between  the  Department  of 
Defense  Index  of  Specifications  and  Standards  (DODISS)  and  the 
Acquisition  Management  Systems  and  Data  Requirements  Control  List 
(AMSDL) . 

15.  The  student  will  understand  the  acquisition  streamlining 
tools  and  techniques  used  to  streamline  acquisition  documents 
(MIL-HDB-248B) . 

16.  The  student  will  understand  the  process  by  which  standards 
and  specifications  are  annotated  to  tailor  out  unnecessary 
requirements  (MIL-HDB-248B) . 

17.  The  student  will  know  the  different  government  software 
packages  available  to  assist  the  Project  Officer  in  the 
preparation  of  acquisition  related  documents. 

18.  The  student  will  demonstrate  the  use  of  the  Acquisition 
Management  Systems  and  Data  Requirements  Control  List  (AMSDL)  and 
Data  Item  Descriptions  (DID's)  to  select  candidate  data  for 
application  as  data  by-products  of  a  SOW  task. 

19.  The  student  will  be  able  to  rewrite  Statement  Of  Work  (SOW) 
paragraphs  by  analyzing  draft  SOW  paragraphs  (for  errors)  and 
data  applications  and  applying  the  appropriate  policies  and 
procedures  to  formulate  a  proper  SOW/CDRL  package. 

20.  The  student  will  be  able  to  properly  generate  new  tailored 
Data  Item  Descriptions  (DIDs)  using  AF  Form  585  (Contract  Data 
Item  Substantiation)  for  a  Statement  of  Work  (SOW) . 

METHOD  OF  INSTRUCTION;  Lecture/Discussion/Group  exercise 

INSTRUCTIONAL  MATERIALS;  Data  Item  Descriptions,  AMSDL,  and 

Viewgraph  Slides 

AUDIO-VIDUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 

INSTRUCTIONAL  EQUIPMENT;  Not  applicable 

REFERENCES ;  Readings/Exercise/Classroom  references 

REQUIRED  STUDENT  PREPARATION;  Lesson  readings 

Read  exercise 


122 


ACQUISITION  PLANNING  AND  ANALYSIS 


SYS  200 

LESSON  TITLE;  Contracting  (II-7) 
TIME;  2.5  hrs 


MODULE  OBJECTIVE (S) :  The  module  objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  know  the  various  parts  of  a  contract. 

2.  The  student  will  understand  the  roles  of  the  different  types 
of  Contracting  Officers  [Contracting  Officer  (PCO) , 

Administrative  Contracting  Officer  (ACO) ,  and  Termination 
Contracting  Officer  (TCO) ]  in  the  systems  acquisition  process. 

3.  The  student  will  understand  the  Source  Selection  Process 
portion  of  the  contracting  process. 

4.  The  student  will  understand  the  acquisition  streamlining 
concepts  applied  to  contracting. 

5.  The  student  will  understand  the  Contract  Administration 
portion  of  the  contracting  process. 

METHOD  OF  INSTRUCTION;  Lecture/ Discuss ion 

INSTRUCTIONAL  MATERIALS ;  Viewgraph  slides 

AUDIO-VIDUAL  AIDS;  Overhead  projectors,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES ;  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  Readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

T.RSSQN  TTTT.F.i  Dealing  with  the  Contractor  (II-8) 
TIME;  1.5  hrs 


MODULE  OBJECTIVE (S) :  The  module  objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  how  a  contractor  is  organized 
to  manage  a  government  acquisition  program. 

2.  The  student  will  understand  contractor  motivation  for  winning 
a  government  contract. 

3.  The  student  will  recognize  the  major  pressures  that  a 
contractor  has  while  managing  a  government  acquisition  program. 

4.  The  student  will  understand  the  proper  ethical/business 
relationship  between  a  contractor  and  the  government. 

5.  The  student  will  recognize  the  best  ways  in  which  the  Project 
Officer  can  improve  the  interface  with  a  contractor. 

6.  The  student  will  understand  the  corrective  action  mechanisms 
for  unsatisfactory  contractor  performance  relative  to  severity. 

7.  The  student  will  understand  the  purpose  of  the  Defense 
Contract  Management  Command. 

METHOD  OF  INSTRUCTION;  Lecture/Discussion 

INSTRUCTIONAL  MATERIALS;  Viewgraph  slides 

AUDIO-VISUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES ;  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  Readings 
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AQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

T.RRsnM  TTTT.Ri  Facilities  Planning  (II-9) 

TIME;  1.5  Hrs 

MODULE  OBJECTIVES (S) :  The  module  objective (s)  for  each  student 
are  as  follows: 

1.  The  student  will  know  the  process  of  acquiring  facilities  for 
systems . 

2.  The  student  will  know  the  services  available  to  Project 
Officers  from  the  acquisition  civil  engineering  organization. 

3.  The  student  Will  know  the  phases  in  facilities  planning. 

4.  The  student  will  know  the  Military  Construction  Program  (MCP) 
planning  process. 

5.  The  student  will  know  how  facilities  requirements  are 
generated . 

6.  The  student  will  identify  the  normal  source  (appropriation)  of 
funds  used  in  facilities  acquisition. 

7.  The  student  will  know  who  is  responsible  for  providing  timely 
facilities  planning  for  a  new  system. 

METHOD  OF  INSTRUCTION;  Lecture/Discussion 

INSTRUCTIONAL  MATERIALS:  Viewgraph  slides 

AUDIO-VIDUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  Applicable 
REFERENCE;  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

LESSON  TITLE!  Program  Control  (III-l) 

TIME;  1  Hrs 

MODULE  OBJECTIVE(S) :  The  Module  Objective(s)  for  each  student  are 
as  follows: 

1.  The  student  will  distinguish  between  the  functions/ 
responsibilities  of  the  Program  Evaluation,  Financial  Management, 
and  Plans  &  Integration  elements  within  Program  Control. 

2.  The  student  will  list  the  typical  end  products  of  the 
Program  Evaluation,  Financial  Management,  and  Plans  &  Integration 
elements  of  Program  Control. 

3.  The  student  will  distinguish  between  appropriation  integrity 
and  fiscal  integrity. 

4.  The  student  will  know  the  different  types  of  budget 
appropriations  /  funds  used  in  Program  Control. 

5.  The  student  will  be  able  to  distinguish  between  the  financial 
terms  of  commitment,  obligation,  and  expenditure. 

6.  The  student  will  know  the  purpose,  and  content  of  the  Selected 
Acquisition  Report  (SAR) . 

7.  The  student  will  know  the  purpose  and  content  of  the 
Defense  Acquisition  Executive  Summary  (DAES) . 

8.  The  student  will  know  the  purpose,  objectives,  and 
requirements  of  the  Acquisition  Program  Baseline  (APB) . 

9.  The  student  will  know  how  Correlation  Matrices  are  used  to 
control  a  program  through  the  acquisition  life  cycle. 

10.  The  student  will  know  the  different  types  of  software  tools 
available  to  assist  the  Project  Officer  in  managing  a  program. 


126 


METHOD  OF  INSTRUCTION;  Lecture/Discussion 
INSTRUCTIONAL  MATERIALS!  Viewgraph  slides 

AUDIO-VIDUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT:  Not  applicable 
REFERENCES :  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  readings 
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ACQUISITION  PLANNING  AND  ANALYSIS 


SYS  200 

LESSON  TITLE;  Cost  Estimating  (III-2) 
TIME;  5  hrs 


MODULE  OBJECTIVE (S) :  The  module  objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  purpose  of  cost  estimating. 

2.  The  student  will  differentiate  between  the  different  types 
of  costs  (Program  Acguisition  Cost,  Ownership  Cost,  Life  Cycle 
Cost,  etc)  encountered  in  acquisition  program  management. 

3.  The  student  will  understand  the  purpose  of  the  different  types 
of  cost  estimates  (Independent  Cost  Analysis,  Independent 
Sufficiency  Review,  Independent  Cost  Study,  etc.). 

4.  The  student  will  identify  the  criteria  used  by  a  cost 
estimator. 

5.  The  student  will  distinguish  between  the  cost  estimating 
methodologies  available  to  perform  cost  estimating. 

6.  The  student  will  understand  the  cost  estimating  technique  of 
economic  escalation. 

7.  The  student  will  understand  the  cost  estimating  technique  of 
learning  curves. 

8.  The  student  will  understand  the  procedures  for  applying  the 
economic  escalation  to  a  cost  estimate. 

9.  The  student  will  understand  the  procedures  for  applying  the 
learning  curve  theory  to  a  cost  estimate. 

10.  The  student  will  differentiate  between  the  two  cost 
estimating  structures. 

11.  The  student  will  understand  the  overall  top  level  procedures 
for  developing  a  cost  estimate  for  a  typical  program. 

12.  The  student  will  develop  a  cost  estimate  for  a  simulated 
acquisition  program. 
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METHOD  OF  INSTRUCTION!  Lecture/Dlscussion 
INSTRUCTIONAL  MATERIALS;  Viewgraph  slides 

AUDIO-VIDUAL  AIDS!  Overhead  projector,  screen,  and  chalkboard 

INSTRUCTIONAL  EQUIPMENT;  Not  applicable 

REFERENCES ;  Exercise  *■  B-IB  Cost  Estimating  Case 

REQUIRED  STUDENT  PREPARATION;  Review  exercise  and  lesson  before 

class 
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ACQUISITION  PLANNING  AND  ANALYSIS 
SYS  200 

T.Ra.qoM  TTTT.Ri  Contractor  Performance  Measurement  (C/SCSC) 

(III-3) 

TIME:  7  hrs 

MODULE  OBJECTIVE (S) :  The  Module  Objective (s)  for  each  student  are 
as  follows: 

1.  The  student  will  understand  the  overall  concept  of  the 
Cost/Schedule  Control  Systems  Criteria  (C/SCSC) . 

2.  The  student  will  understand  the  objectives  of  the  Cost/ 
Schedule  Control  Systems  Criteria  (C/SCSC) . 

3.  The  student  will  understand  the  purpose  of  the  Performance 
Measurement  Baseline  (PMB) . 

4.  The  student  will  understand  the  procedures  for  developing 
a  Performance  Measurement  Measurement  (PMB) . 

5.  The  student  will  understand  the  concept  of  variance  analysis 
as  it  applies  to  Contractor  Performance  Measurement  (C/SCSC) . 

6.  The  student  will  understand  the  key  concepts  (BCWS,  BCWP, 

ACWP,  Earned  Value,  BAC,  PMB,  MR,  EAC,  CV,  SV,  CPI,  SPI,  CV%, 

SV%,  %  Complete,  %  Spent,  %  Scheduled,  and  TCPI)  associated  with 
analyzing  contractor  performance  data. 

7.  The  student  will  understand  the  difference  between  a  Cost 
Performance  Report  (CPR) ,  a  Cost/Schedule  Status  Report  (CSSR) , 
and  a  Contract  Funds  Status  Report  (CFSR) . 

8.  The  student  will  calculate  the  following:  CV,  SV,  EAC,  CPI, 
SPI,  CV%,  SV%,  %  Complete,  %  Spent,  %  Scheduled,  VC,  and  TCPI. 

METHOD  OF  INSTRUCTION:  Lecture/Discussion 

INSTRUCTIONAL  MATERIALS;  Viewgraph  slides 

AUDIO-VISUAL  AIDS;  Overhead  projector,  screen,  and  chalkboard 
INSTRUCTIONAL  EQUIPMENT;  Not  applicable 
REFERENCES;  Readings 

REQUIRED  STUDENT  PREPARATION;  Lesson  Readings 
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Appendix  B:  Self-Efficacy  Questionnaires 

SYSaQQ  QBJKCTIVgS  RKVIMt 


Purpoaa 

The  purpose  of  this  questionnaire  is  to  help  AFIT  identify  strengths, 
weaknesses,  and  improvement  areas  in  its  SYS200  "Acquisition  Planning 
and  Analysis"  course.  This  survey  is  part  of  an  ongoing  research 
effort,  and  AFIT  will  use  your  honest,  objective  inputs  to  design 
improved  programs  for  future  students . 

Anonymltv 

Your  name  will  not  be  used  or  associated  with  your  answers  on  this 
survey.  Your  social  security  number  is  requested  only  as  a  means  of 
tracking  your  answer  sheet  for  a  follow-on  questionnaire.  AFIT  will  not 
use  or  show  your  social  security  number  in  any  other  manner. 

Reaulta 

The  results  of  this  research  will  be  published  in  an  AFIT  thesis  in 
September,  1992,  and  the  final  report  will  be  permanently  stored  with 
the  Defense  Technical  Information  Center  (DTIC) .  As  a  participant  in 
this  survey,  you  have  an  opportunity  to  obtain  a  summary  of  the  findings 
and  conclusions.  The  instructor  will  make  available  a  sign-up  sheet  on 
which  you  may  indicate  your  desire  to  receive  a  summary. 

Inafcruetiona 

Use  only  a  number  2  pencil  on  the  answer  sheet. 

DO  NOT  put  your  name  on  the  answer  sheet  or  questionnaire.  Mark  only 
your  Social  Security  Account  Number  (SSAN)  in  the  boxes  provided. 

Answer  all  questions  according  to  your  initial  reaction.  Please  do  not 
change  your  responses  to  earlier  questions  based  on  information 
presented  in  later  questions. 

IF  YOU  ARE  SOMEWHAT  UNSURE  OF  YOUR  ANSWER,  USE  YOUR  FIRST  "INSTINCTIVE" 
RESPONSE.  IF  THE  QUESTION  SEEMS  CONFUSING  OR  AMBIGUOUS,  T.gAvie  thf. 
AMSWBR  BIAMK  AND  INDICATE  WHY  YOU  THINK  SO  BY  COMMENTING  DIRECTLY  ON  THE 
QUESTIONNAIRE  THE  ANSWER  SHEET)  IN  WHATEVER  SPACE  IS  AVAILABLE. 

This  questionnaire  has  been  designed  to  take  approximately  20-25 
minutes.  Do  not  spend  too  much  time  on  any  particular  question. 

Once  you  have  completed  this  questionnaire,  return  it  together  with  your 
answer  sheet  to  the  instructor. 
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Please  use  the  following  responses  in  answering  questions  1  through  99. 


a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


1.  The  Advanced  Tactical  Fighter  (ATF)  program  has  completed  the 
Demonstration  and  Validation  phase.  I  can  name  which  milestone  the 
program  has  reached  as  it  transitions  to  the  next  phase. 

2.  If  told  that  the  C-17  program  recently  began  Low  Rate  Initial 
Production,  I  could  identify  which  phase  of  the  acquisition  life  cycle 
the  program  is  in. 

3.  I  could  describe  the  purpose  of  the  Defense  Accjuisition  Board  (DAB) 
to  a  newly  assigned  co-worker. 

4.  I  could  describe  the  differences  between  a  Mission  Need  Statement 
(MNS)  and  an  Operational  Requirements  Document  (ORD) . 

5.  I  can  state  the  purpose  of  the  Biennial  Planning,  Programming,  and 
Budgeting  System  (BPPBS) . 

6.  I  can  name  the  position  of  the  individual  to  whom  the  six  Air  Force 
Program  Executive  Officers  (PEOs)  report. 

7.  Given  a  budget  estimate  for  an  acquisition  program,  I  could 
determine  the  appropriate  Acquisition  Category  (i.e.,  ACAT  ID,  IC,  II, 
III,  or  IV)  to  which  it  most  likely  belongs. 

8.  I  can  differentiate  among  Acquisition  Cost,  Ownership  Cost,  and 
Total  Life  Cycle  Cost. 

9.  If  requested,  I  could  develop  a  cost  estimate  in  preparation  for  a 
budget  submission. 

10.  I  could  apply  "learning  curve"  theory  to  a  cost  estimate. 

11.  I  can  distinguish  between  various  cost  estimating  methodologies. 

12.  I  could  state  the  purpose  of  an  Independent  Cost  Analysis  (ICA) . 

13.  I  can  describe  the  differences  in  content  between  a  Cost 
Performance  Report  (CPR)  and  a  Cost/Schedule  Status  Report  (CSSR) . 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


14.  I  can  state  the  purpose  of  the  Performance  Measurement  Baseline. 

15.  I  could  describe  the  overall  concept  of  the  Cost/Schedule  Control 
Systems  Criteria  (C/SCSC) . 

16.  Given  both  budgeted  and  actual  cost  and  schedule  data,  I  could 
calculate  cost  variance  and  schedule  variance. 

17.  I  can  identify  the  benefits  of  proper  Acquisition  Logistics 
planning. 

18.  I  can  describe  the  Logistics  Support  Analysis  (LSA)  process  used  to 
establish  the  logistics  support  structure. 

19.  I  could  define  the  objectives  of  Integrated  Logistics  Support  (ILS) 
for  a  newly  assigned  co-worker. 

20.  I  can  describe  the  ten  DoD  elements  of  logistics  (i.e., 

Maintenance,  Supply  Support,  etc.). 

21.  I  could  discuss  the  concept  of  Acquisition  Logistics  planning  as  it 
pertains  to  acquisition  management. 

22.  I  can  describe  the  major  functions  of  Systems  Engineering  (i.e.. 
Functional  Analysis,  Synthesis,  Evaluation,  and  Description) . 

23.  I  can  identify  the  iterative  steps  of  the  Systems  Engineering 
process . 

24.  I  can  define  the  five  types  of  systems  specifications  (Types  A,  B, 
C,  D,  and  E) . 

25.  I  can  state  the  purpose  of  a  Functional  Configuration  Audit. 

26.  I  can  state  the  purpose  of  a  Preliminary  Design  Review  (PDR) . 

27.  Given  a  descriptive  summary  of  a  project,  I  could  develop  a  draft 
Work  Breakdown  Structure  (WBS) . 

28.  I  can  list  the  benefits  of  using  WBS  in  acquisition  management. 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


29.  I  can  describe  the  relationship  between  WBS  and  contractor 
performance  measurement . 

30.  If  requested,  I  could  describe  the  relationship  between  a  WBS  and 
the  content  of  a  Statement  of  Work  (SOW) . 

31.  I  can  state  the  purposes  of  the  WBS  in  acquisition  planning. 

32 .  I  can  identify  the  current  changes  in  DoD  manufacturing  and  the 
problems  associated  with  implementing  these  changes. 

33.  I  could  discuss  the  relative  global  competitiveness  of  the  U.S. 
manufacturing  industry. 

34.  I  can  describe  the  role  of  manufacturing  in  the  DoD  systems 
acquisition  process. 

35.  I  can  define  a  Hardware  Configuration  Item  (HWCI)  and  a  Computer 
Software  Configuration  Item  (CSCI) . 

36.  I  could  distinguish  among  the  three  types  of  configuration 
management  baselines  (functional,  allocated,  and  product) . 

37.  I  can  describe  the  four  functions  of  Configuration  Management 
(i.e..  Configuration  Identification,  Configuration  Control,  Status 
Accounting,  and  Configuration  Audits) . 

38.  I  can  identify  the  congressional  committees  involved  in  the  DoD 
budget  enactment  process . 

39.  I  could  define  what  a  Program  Element  is. 

40.  I  can  state  the  purpose  of  the  Chairman's  Program  Assessment  (CPA) . 

41.  I  could  explain  the  differences  between  authorizations  and 
appropriations . 

42 .  I  can  describe  the  role  of  the  Defense  Planning  Guidance  (DPG)  in 
the  Biennial  Planning,  Programming,  and  Budgeting  System  (BPPBS) . 

43.  I  can  explain  the  various  uses  of  different  appropriations  (for 
example,  3600,  3400,  or  3080) . 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Coir^letely 


44.  I  can  describe  the  process  of  committing  and  obligating  funds. 

45.  I  could  discuss  the  steps  involved  in  the  preparation  of  a  Request 
for  Proposal  (RFP) . 

46.  An  acquisition  agency  is  developing  its  acquisition  strategy  for  a 
new  research  and  development  program.  I  can  name  the  type  of  contract 
which  the  new  program  is  most  likely  to  use . 

47.  If  assigned  to  a  management  team  for  a  new  acquisition  project,  I 
could  draft  an  outline  of  the  new  project's  acquisition  plan. 

48.  I  can  identify  the  sequence  of  activities  associated  with  planning 
for  a  source  selection. 

49.  I  could  discuss  the  reasons  for  which  "other  than  full  and  open 
competition"  may  be  justified. 

50.  I  can  describe  the  "waterfall"  process  of  software  development 
outlined  in  DoD-STD-2167A. 

51.  I  can  state  the  purpose  of  a  Conputer  Resources  Working  Group 
(CRWG)  . 

52.  I  could  explain  the  differences  between  the  acquisition  management 
of  Information  Systems  Resources  (ISR)  and  Mission  Critical  Computer 
Resources  (MCCR) . 

53.  I  can  identify  the  phases  of  the  software  acquisition  life  cycle. 

54.  Given  a  set  of  requirements,  I  could  distinguish  between  those 
which  belong  in  the  Statement  of  Work  (SOW) ,  the  Contract  Data 
Requirements  List  (CDRL) ,  and  the  Contract  Specification. 

55.  I  can  state  the  purpose  of  a  specification. 

56.  If  requested,  I  could  write  a  Statement  of  Work  (SOW) . 

57.  I  could  discuss  potential  problems  caused  by  a  poorly  written 
Statement  of  Work  (SOW) . 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e'  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


58.  I  can  distinguish  between  a  specification  and  a  standcrd. 

59.  I  can  differentiate  between  Qualification  and  Acceptance  testing. 

60.  I  could  describe  the  process  used  to  "tailor  a  spec"  to  meet 
specific  acquisition  requirements. 

61.  I  can  interpret  the  information  presented  by  different  project 
scheduling  formats  (for  example,  a  Gantt  chart,  a  PERT  chart,  or  a 
Line-of-Balance  diagram) . 

62.  I  can  build  a  project  network  schedule. 

63.  I  can  describe  the  relative  strengths  and  weaknesses  of  the  various 
scheduling  techniques . 

64.  I  could  use  an  Integrated  Program  Master  Schedule  to  assess  the 
status  of  a  particular  program. 

65.  I  can  determine  a  network's  critical  path. 

66.  I  can  identify  strengths  and  weaknesses  of  the  various 
organizational  structures  (i.e.,  functional,  project,  matrix,  etc.) 

67.  I  could  describe  the  various  roles  and’  responsibilities  of  program 
management  to  a  newly  assigned  co-worker. 

68.  I  can  describe  the  unique  aspects  of  defense  systems  acquisition 
management  which  distinguish  it  from  other  forms  of  management. 

69.  I  can  define  the  following  financial  terms:  commitment,  obligation, 
and  expenditure . 

70.  I  could  explain  the  various  functions  of  Program  Control  (program 
evaluation,  financial  management,  and  plans  &  integration) . 

71.  I  can  describe  the  steps  involved  in  the  Source  Selection  Process. 

72.  I  could  distinguish  among  the  various  roles  and  responsibilities  of 
the  Principal  Contracting  Officer  (PCO) ,  the  Administrative  Contracting 
Officer  (ACO) ,  and  the  Terminating  Contracting  Officer  (TCO) . 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Con^letely 


73.  I  could  describe  the  various  sections  of  the  Uniform  Contract 
Format  (for  example.  Section  A,  The  Schedule;  Section  L,  Instructions  to 
Offerors;  etc. ) . 

74.  I  can  discuss  the  applicability  of  various  types  of  contracts  (for 
example.  Firm  Fixed  Price  (FFP) ,  Cost  Plus  Fixed  Fee  (CPFF) ,  etc.). 

75.  I  can  describe  corrective  action  mechanisms  available  to 
acquisition  managers  for  dealing  with  unsatisfactory  contractor 
performance . 

76.  I  could  discuss  ethical  considerations  in  dealings  between  the 
government  and  contractors. 

77.  I  can  describe  the  mission  of  the  Defense  Contract  Management 
Command  (DCMC) . 

78.  I  could  describe  the  roles  and  responsibilities  of  a  Defense  Plant 
Representative  Office  (DPRO)  in  administering  a  contract. 

79.  I  can  describe  the  Military  Construction  Program  (MCP)  as  it 
pertains  to  facilities  acquisition  management. 

80.  I  could  identify  the  various  phases  of  facilities  planning. 

81.  I  can  explain  how  facilities  requirements  are  generated. 

82.  I  can  define  the  role  of  Test  and  Evaluation  (T&E)  in  acquisition 
management . 

83.  I  can  state  the  purpose  of  the  Requirements  Correlation  Matrix 
(RCM) . 

84.  I  could  describe  the  differences  among  Developmental  Test  and 
Evaluation  (DT&E) ,  Initial  Operational  Test  and  Evaluation  (lOT&E),  and 
Follow-on  Operational  Test  and  Evaluation  (FOT&E) . 


Questions  85-99  pertain  to  the  Acquisition  Professional  Development 
Program  (APDP) .  Successful  completion  of  SYS200  satisfies  one  of  the 
course  requirements  for  APDP  certification. 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Coit^letely 


85.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Acquisition  Logistics  functional  discipline. 

86.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Conmunications/Coaputexs  functional  discipline. 

87.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Comptrollar  functional  discipline. 

88.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Contracting  functional  discipline. 

89.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Developmental  Engineering  functional  discipline. 

90.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Program  Management  functional  discipline. 

91.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 

Manufacturing/Quality  Assurance  functional  discipline. 

92.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Scientific  6  Technical  functional  discipline. 

93.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Test  6  Evaluation  functional  discipline. 

94.  I  can  describe  the  purpose  of  the  Acquisition  Professional 
Development  Program  (APDP) . 

95.  I  could  list  my  functional  discipline  certification  requirements 
under  APDP . 

96.  I  believe  these  certification  requirements  are  appropriate  for 
acquisition  managers  in  my  functional  discipline. 

97.  I  believe  these  certification  requirements  are  achievable. 

98.  I  believe  I  will  have  adecjuate  opportunities  for  achieving  these 
certification  requirements. 

99.  SyS200  should  be  a  required  course  for  acquisition  managers  in  my 
functional  discipline. 
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Questions  100,  101,  and  102.  Please  select  only  ONE  response  from 
Questions  100-102  combined.  Do  not  select  more  than  one  answer. 


My  primary  ftinctlonal  discipline  or  area  of  eipertise  is: 

(Note:  If  you  are  currently  seeking  or  planning  to  seek  certification 
in  the  Acquisition  Professional  Development  Program  (APDP) ,  you 
should  respond  by  selecting  the  primary  APDP  functional 
discipline  for  which  you  intend  to  seek  certification. 

APDP  functional  disciplines  are  indicated  by  the  asterisks . ) 

100.  a)  Acquisition  Logistics  * 

b)  Communications /Computers  * 

c)  Comptroller  * 

d)  Contracting  * 

e)  Developmental  Engineering  * 

101.  a)  Manufacturing/Quality  Assurance  * 

b)  Operations 

c)  Program  Management  * 

d)  Requirements 

e)  Scientific/Technical  * 

102.  a)  Test  &  Evaluation  * 

b)  Not  listed  here.  Please  code  in  102(b)  on  the  answer  sheet  and 

write  your  functional  discipline  here _ 

and  your  Social  Security  Number  here _ . 


Questions  103,  104,  105,  and  106.  Please  select  only  ONE  response  from 
Questions  103-106  combined.  Do  not  select  more  than  one  answer. 


The  nuinber  of  years  of  acq;uisibion  experience  Z  have  (rounded  up  to  the 
nearest  whole  nundber)  is: 


103.  a) 

0 

104.  a) 

5 

105.  a) 

10 

b) 

1 

b) 

6 

b) 

11 

c) 

2 

c) 

7 

c) 

12 

d) 

3 

d) 

8 

d) 

13 

e) 

4 

e) 

9 

e) 

14 

106.  a)  15  or  more 


Please  use  the  space  on  the  back  of  this  page  for  any  comments  you  may 
have  about  this  questionnaire. 

Thank  you  for  your  time  and  effort. 
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SYS2Q0  POST-COURSg  OBJECTIVgS  RgVIBW 


Purpcaa 

The  purpose  of  this  questionnaire  is  to  help  AFIT  identify  strengths, 
weaknesses,  and  improvement  areas  in  its  SYS200  "Acquisition  Planning 
and  Analysis"  course. 

Participation 

Your  participation  is  voluntary.  This  survey  is  part  of  an  important 
research  effort  and  AFIT  will  use  your  honest,  objective  inputs  to 
design  improved  programs  for  future  students. 

AnonymitY 

Your  name  will  not  be  used  or  associated  with  your  answers  on  this 
survey,  or  on  the  earlier  pre-course  survey.  Your  social  security 
number  is  requested  only  as  a  means  of  matching  your  answer  sheet  to 
your  responses  from  the  pre-course  survey.  AFIT  will  not  use  or  show 
your  social  security  number  in  any  other  manner. 

Reaulta 

The  results  of  this  research  will  be  published  in  an  AFIT  thesis  in 
September,  1992,  and  the  final  report  will  be  permanently  stored  with 
the  Defense  Technical  Information  Center  (DTIC) .  As  a  participant  in 
this  survey,  you  have  an  opportunity  to  obtain  a  summary  of  the  findings 
and  conclusions.  The  instructor  will  make  available  a  sign-up  sheet  on 
which  you  may  indicate  your  desire  to  receive  a  summary. 

Inatcttctiona 

Use  only  a  number  2  pencil  on  the  answer  sheet . 

DO  NOT  put  your  name  on  the  answer  sheet  or  questionnaire.  Mark  only 
your  Social  Security  Account  Number  (SSAN)  in  the  boxes  provided. 

Answer  all  questions  according  to  your  initial  reaction.  Please  do  not 
change  your  responses  to  earlier  questions  based  on  information 
presented  in  later  questions. 

IF  YOU  ARE  SOMEWHAT  UNSURE  OF  YOUR  ANSWER,  USE  YOUR  FIRST  "INSTINCTIVE" 
RESPONSE.  IF  THE  QUESTION  SEEMS  CONFUSING  OR  AMBIGUOUS,  t.raw  thit. 
ANSWER  BLANK  AND  INDICATE  WHY  YOU  THINK  SO  BY  COMMENTING  DIRECTLY  ON  THE 
QUESTIONNAIRE  (UQI  THE  ANSWER  SHEET)  IN  WHATEVER  SPACE  IS  AVAILABLE. 

This  questionnaire  has  been  designed  to  take  approximately  20-25 
minutes.  Do  not  spend  too  much  time  on  any  particular  question. 

Once  you  have  completed  this  questionnaire,  please  keep  it  and  your 
answer  sheet  together  and  return  them  both  to  the  instructor. 
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Please  use  the  following  responses  in  answering  questions  1  through  99. 


a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


1.  I  could  describe  the  differences  among  Developmental  Test  and 
Evaluation  (DT&E) ,  Initial  Operational  Test  and  Evaluation  (lOT&E),  and 
Follow-on  Operational  Test  and  Evaluation  (FOT&E) . 

2 .  I  can  state  the  purpose  of  the  Requirements  Correlation  Matrix 
(RCM)  . 

3.  I  can  define  the  role  of  Test  and  Evaluation  (T&E)  in  acquisition 
management . 

4.  I  can  explain  how  facilities  requirements  are  generated. 

5.  I  could  identify  the  various  phases  of  facilities  planning. 

6.  I  can  describe  the  Military  Construction  Program  (MCP)  as  it 
pertains  to  facilities  acquisition  management. 

I .  I  could  describe  the  roles  and  responsibilities  of  a  Defense  Plant 
Representative  Office  (DPRO)  in  administering  a  contract. 

8.  I  can  describe  the  mission  of  the  Defense  Contract  Management 
Command  (DCMC) . 

9.  I  could  discuss  ethical  considerations  in  dealings  between  the 
government  and  contractors. 

10.  I  can  describe  corrective  action  mechanisms  available  to 
acquisition  managers  for  dealing  with  unsatisfactory  contractor 
performance . 

II.  I  can  discuss  the  applicability  of  various  types  of  contracts  (for 
example.  Firm  Fixed  Price  (FFP) ,  Cost  Plus  Fixed  Fee  (CPFF) ,  etc.) . 

12.  I  could  describe  the  various  sections  of  the  Uniform  Contract 
Format  (for  example.  Section  A,  The  Schedule;  Section  L,  Instructions  to 
Offerors;  etc. ) . 


141 


a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


13.  I  could  distinguish  among  the  various  roles  and  responsibilities  of 
the  Principal  Contracting  Officer  (PCO) ,  the  Administrative  Contracting 
Officer  (ACO) ,  and  the  Terminating  Contracting  Officer  (TCO) . 

14.  I  can  describe  the  steps  involved  in  the  Source  Selection  Process. 

15.  I  could  explain  the  various  functions  of  Program  Control  (program 
evaluation,  financial  management,  and  plans  &  integration) . 

16.  I  can  define  the  following  financial  terms:  commitment,  obligation, 
and  expenditure . 

17.  I  can  describe  the  unique  aspects  of  defense  systems  acquisition 
management  which  distinguish  it  from  other  forms  of  management. 

18.  I  could  describe  the  various  roles  and  responsibilities  of  program 
management  to  a  newly  assigned  co-worker. 

19.  I  can  identify  strengths  and  weaknesses  of  the  various 
organizational  structures  (i.e.,  functional,  project,  matrix,  etc.) 

20.  I  can  determine  a  network's  critical  path. 

21.  I  could  use  an  Integrated  Program  Master  Schedule  to  assess  the 
status  of  a  particular  program. 

22 .  I  can  describe  the  relative  strengths  and  weaknesses  of  the  various 
scheduling  techniques. 

23.  I  can  build  a  project  network  schedule. 

24.  I  can  interpret  the  information  presented  by  different  project 
scheduling  formats  (for  example,  a  Gantt  chart,  a  PERT  chart,  or  a 
Line-of-Balance  diagram) . 

25.  I  could  describe  the  process  used  to  "tailor  a  spec"  to  meet 
specific  acquisition  requirements. 

26.  I  can  differentiate  between  Qualification  and  Acceptance  testing. 

27.  I  can  distinguish  between  a  specification  and  a  standard. 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


28.  I  could  discuss  potential  problems  caused  by  a  poorly  written 
Statement  of  Work  (SOW) . 

29.  If  requested,  I  could  write  a  Statement  of  Work  (SOW) . 

30.  I  can  state  the  purpose  of  a  specification. 

31.  Given  a  set  of  requirements,  I  could  distinguish  between  those 
which  belong  in  the  Statement  of  Work  (SOW),  the  Contract  Data 
Requirements  List  (CDRL) ,  and  the  Contract  Specification. 

32.  I  can  identify  the  phases  of  the  software  acquisition  life  cycle. 

33.  I  could  explain  the  differences  between  the  acquisition  management 
of  Information  Systems  Resources  (ISR)  and  Mission  Critical  Computer 
Resources  (MCCR) . 

34.  I  can  state  the  purpose  of  a  Computer  Resources  Working  Group 
(CRWG)  . 

35.  I  can  describe  the  "waterfall"  process  of  software  development 
outlined  in  DoD-STD-2167A. 

36.  I  could  discuss  the  reasons  for  which  "other  than  full  and  open 
competition"  may  be  justified. 

37.  I  can  identify  the  sequence  of  activities  associated  with  planning 
for  a  source  selection. 

38.  If  assigned  to  a  management  team  for  a  new  acquisition  project,  I 
could  draft  an  outline  of  the  new  project's  acquisition  plan. 

39.  An  acquisition  agency  is  developing  its  acquisition  strategy  for  a 
new  research  and  development  program.  I  can  name  the  type  of  contract 
which  the  new  program  is  most  likely  to  use. 

40.  I  could  discuss  the  steps  involved  in  the  preparation  of  a  Request 
for  Proposal  (RFP) . 

41.  I  can  describe  the  process  of  committing  and  obligating  funds. 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Conpletely 


42.  I  can  explain  the  various  uses  of  different  appropriations  (for 
example,  3600,  3400,  or  3080) . 

43.  I  can  describe  the  role  of  the  Defense  Planning  Guidance  (DPG)  in 
the  Biennial  Planning,  Programming,  and  Budgeting  System  (BPPBS) . 

44.  I  could  explain  the  differences  between  authorizations  and 
appropriations . 

45.  I  can  state  the  purpose  of  the  Chairman's  Program  Assessment  (CPA). 

46.  I  could  define  what  a  Program  Element  is. 

47 .  I  can  identify  the  congressional  committees  involved  in  the  DoD 
budget  enactment  process . 

48.  I  can  describe  the  four  functions  of  Configuration  Management 
(i.e..  Configuration  Identification,  Configuration  Control,  Status 
Accounting,  and  Configuration  Audits) . 

49.  I  could  distinguish  among  the  three  types  of  configuration 
management  baselines  (functional,  allocated,  and  product) . 

50.  I  can  define  a  Hardware  Configuration  Item  (HWCI)  and  a  Computer 
Software  Configuration  Item  (CSCI) . 

51.  I  can  describe  the  role  of  manufacturing  in  the  DoD  systems 
acquisition  process. 

52.  I  could  discuss  the  relative  global  competitiveness  of  the  U.S. 
manufacturing  industry. 

53.  I  can  identify  the  current  changes  in  DoD  manufacturing  and  the 
problems  associated  with  implementing  these  changes. 

54.  I  can  state  the  purposes  of  a  WorJc  Brea)cdown  Structure  (WBS)  in 
acquisition  planning. 

55.  If  requested,  I  could  describe  the  relationship  between  a  WBS  and 
the  content  of  a  Statement  of  Worlc  (SOW)  . 

56.  I  can  describe  the  relationship  between  WBS  and  contractor 
performance  measurement . 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


57.  I  can  list  the  benefits  of  using  a  Work  Breakdovm  Structure  (WBS) 
in  acquisition  management. 

58.  Given  a  descriptive  summary  of  a  project,  I  could  develop  a  draft 
Work  Breakdovm  Structure  (WBS) . 

59.  I  can  state  the  purpose  of  a  Preliminary  Design  Revievf  (PDR) . 

60.  I  can  state  the  purpose  of  a  Functional  Configuration  Audit. 

61.  I  can  define  the  five  types  of  systems  specifications  (Types  A,  B, 
C,  D,  and  E) . 

62.  I  can  identify  the  iterative  steps  of  the  Systems  Engineering 
process . 

63.  I  can  describe  the  major  functions  of  Systems  Engineering  (i.e.. 
Functional  Analysis,  Synthesis,  Evaluation,  and  Description) . 

64.  I  could  discuss  the  concept  of  Acquisition  Logistics  planning  as  it 
pertains  to  acquisition  management. 

65.  I  can  describe  the  ten  DoD  elements  of  logistics  (i.e.. 

Maintenance,  Supply  Support,  etc.). 

66.  I  could  define  the  objectives  of  Integrated  Logistics  Support  (ILS) 
for  a  nevfly  assigned  co-v/orker. 

67.  I  can  describe  the  Logistics  Support  Analysis  (LSA)  process  used  to 
establish  the  logistics  support  structure. 

68.  I  can  identify  the  benefits  of  proper  Acquisition  Logistics 
planning. 

69.  Given  both  budgeted  and  actual  cost  and  schedule  data,  I  could 
calculate  cost  variance  and  schedule  variance. 

70.  I  could  describe  the  overall  concept  of  the  Cost/Schedule  Control 
Systems  Criteria  (C/SCSC) . 

71.  I  can  state  the  purpose  of  the  Performance  Measurement  Baseline. 

72.  I  can  describe  the  differences  in  content  betv^een  a  Cost 
Performance  Report  (CPR)  and  a  Cost/Schedule  Status  Report  (CSSR) . 

73.  I  could  state  the  purpose  of  an  Independent  Cost  Analysis  (ICA) . 
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a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 


74.  I  can  distinguish  between  various  cost  estimating  methodologies. 

75.  I  could  apply  "learning  curve"  theory  to  a  cost  estimate. 

76.  If  requested,  I  could  develop  a  cost  estimate  in  preparation  for  a 
budget  submission. 

77.  I  can  differentiate  among  Acquisition  Cost,  Ownership  Cost,  and 
Total  Life  Cycle  Cost . 

78.  Given  a  budget  estimate  for  an  acquisition  program,  I  could 
determine  the  appropriate  Acquisition  Category  (i.e.,  ACAT  ID,  IC,  II, 
III,  or  IV)  to  which  it  most  likely  belongs. 

79.  I  can  name  the  position  of  the  individual  to  whom  the  six  Air  Force 
Program  Executive  Officers  (PEOs)  report. 

80.  I  can  state  the  purpose  of  the  Biennial  Planning,  Programming,  and 
Budgeting  System  (BPPBS) . 

81.  I  could  describe  the  differences  between  a  Mission  Need  Statement 
(MNS)  and  an  Operational  Requirements  Document  (ORD) . 

82.  1  could  describe  the  purpose  of  the  Defense  Acquisition  Board  (DAB) 
to  a  newly  assigned  co-worker. 

83.  If  told  that  the  C-17  program  recently  began  Low  Rate  Initial 
Production,  I  could  identify  which  phase  of  the  accjuisition  life  cycle 
the  program  is  in. 

84.  The  Advanced  Tactical  Fighter  (ATF)  program  has  completed  the 
Demonstration  and  Validation  phase.  I  can  name  which  milestone  the 
program  has  reached  as  it  transitions  to  the  next  phase. 


Questions  85-99  pertain  to  the  Acquisition  Professional  Development 
Program  (APDP) .  As  stated  in  AFR  36-27,  "APDP  is  based  on  the 
functional  responsibility  for  management  of  specific  acquisition 
disciplines...  The  Air  Force  functional  manager  for  each  discipline  has 
determined  the  unique  acquisition  education,  training,  and  experience 
needed  to  perform  acquisition  duties  in  advancing  levels  of  complexity 
and  responsibility." 

In  keeping  with  provisions  of  the  Defense  Acquisition  Workforce 
Improvement  Act  and  applicable  DoD  regulations,  APDP  certification  is 
required  for  Air  Force  acquisition  managers.  Successful  completion  of 
SYS200  satisfies  one  of  the  course  requirements  for  APDP  certification. 
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a)  Do  Not 

b)  Agree 

c)  Agree 

d)  Agree 

e )  Agree 

Agree 

Slightly 

Moderately 

Considerably 

Completely 

85.  SYS200  should  be  a  required  course  for  acquisition  managers  in  my 
functional  discipline. 

86.  I  can  describe  the  purpose  of  the  Acquisition  Professional 
Development  Program  (APDP) . 

87 .  I  could  list  my  functional  discipline  certification  requirements 
under  APDP . 

88.  I  believe  these  certification  requirements  are  appropriate  for 
acquisition  managers  in  my  functional  discipline. 

89.  I  believe  these  certification  requirements  are  achievable. 

90.  I  believe  I  will  have  adequate  opportunities  for  achieving  these 
certification  requirements. 

91.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Test  &  Evaluation  functional  discipline. 

92 .  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Scientific  &  Technical  functional  discipline. 

93.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Manufacturing/Quality  Assurance  functional  discipline. 

94.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Program  Management  functional  discipline. 

95.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 

Developmental  Engineering  functional  discipline. 

96.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Contracting  functional  discipline. 

97 .  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Conptroller  functional  discipline. 

98.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Communications/Computers  functional  discipline. 

99.  I  can  describe  the  acquisition  roles  and  responsibilities  of  the 
Acquisition  Logistics  functional  discipline. 
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Questions  100,  101,  and  102.  Please  select  only  ONE  response  from 
Questions  100-102  combined.  Do  not  select  more  than  one  answer. 


primary  functional  discipline  or  area  of  expertise  is: 

(Note:  If  you  are  currently  seeking  or  planning  to  seek  certification 
in  the  Acquisition  Professional  Development  Program  (APDP) ,  you 
should  respond  by  selecting  the  primary  APDP  functional 
discipline  for  which  you  intend  to  seek  certification. 

APDP  functional  disciplines  are  indicated  by  the  asterisks . ) 

100.  a)  Acquisition  Logistics  * 

b)  Communications/Computers  * 

c)  Comptroller  * 

d)  Contracting  * 

e)  Developmental  Engineering  * 

101.  a)  Manufacturing/Quality  Assurance  * 

b)  Operations 

c)  Program  Management  * 

d)  Requirements 

e)  Scientific/Technical  * 

102.  a)  Test  &  Evaluation  * 

b)  Not  listed  here.  Please  code  in  102(b)  on  the  answer  sheet  and 

write  your  functional  discipline  here _ 

and  your  Social  Security  Number  here _ . 


Questions  103,  104,  105,  and  106.  Please  select  only  ONE  response  from 
Questions  103-106  combined .  Do  not  select  more  than  one  answer. 


The  number  of  years  of  acquisition  experience 


nearest 

whole 

number) 

is: 

103.  a) 

0 

104.  a) 

5 

105.  a) 

10 

b) 

1 

b) 

6 

b) 

11 

c) 

2 

c) 

7 

c) 

12 

d) 

3 

d) 

8 

d) 

13 

e) 

4 

e) 

9 

e) 

14 

I  have  (rounded  up  to  the 


106.  a)  15  or  more 


Please  use  the  space  on  the  back  of  this  page  for  any  comments  you  may 
have  about  this  questionnaire. 

Thank  you  for  your  time  and  effort. 
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Appendix  C; _ Supervisor  Survey  Questionnaire 


INSTRUCTIONS:  Please  circle  the  appropriate  response  to  each  question. 
Return  by  pouch  or  regular  mail  to:  AFIT/LSY  (Attn:  Capt  Hill) 

Wright-Patterson  AFB,  OH  45433 

1.  Did  you  supervise  the  individual  prior  to  his/her  attendance  at 
SYS  200? 

a.  Yes 

b.  No  (please  skip  to  Question  6) 

2.  Before  attending  SYS  200,  my  subordinate's  knowledge  of  acquisition 
management  was  satisfactory. 

a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 

3.  Since  attending  SYS  200,  my  subordinate’s  knowledge  of  acquisition 
management  has  improved. 

a)  Do  Not  b)  Agree  c)  Pgree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 

4.  Since  attending  SYS  200,  my  subordinate's  job-related  confidence  has 

improved. 

a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 

5.  Since  attending  SYS  200,  my  subordinate's  current  job  performance 

has  improved. 

a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 

6.  Since  attending  SYS  200,  my  subordinate  has  been  able  to  use  the 

information  presented  in  the  course. 

a)  Do  Not  b)  Agree  c)  Agree  d)  Agree  e)  Agree 

Agree  Slightly  Moderately  Considerably  Completely 

7 .  Please  provide  any  comments  or  suggestions  which  will  assist  in 
course  improvement . 


Thank  you  for  your  time  and  effort. 


FD  1  Acquisition  Logistics 

FD  2  Communications /Computers 

FD  3  Financial  Management  (Comptroller) 

FD  4  Contracting 

FD  5  Developmental  Engineering  (combined  with  FD  8) 

FD  6  Manufacturing/Quality  Assurance 

FD  7  Program  Management 

FD  8  Science  &  Technology 

FD  9  Test  &  Evaluation 

FD  10  Other  (APDP  Non-participant) 

FD  11  Logistics  (Sustainment-oriented  subgroup) 


PT  Knowledge  Measure  PreTest 

QAVG  Knowledge  Measure  Posttest  (Quiz  Average) 

DELTA  Knowledge  Increase  (Posttest  minus  pretest) 


PRETOT  Self-Efficacy  Measure  Pretest 

POSTTOT  Self-Efficacy  Measure  Posttest 

DELTA  Self-Efficacy  Increase  (Posttest  minus  pretest) 
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KRUSKAL-WALLIS  ONE-WAY  NONPARAMETRIC  AOV  FOR  PT  BY  FD 


MEAN 

SAMPLE 

FD 

RANK 

SIZE 

1 

64.1 

24 

2 

48.8 

3 

3 

45.7 

9 

4 

52.5 

3 

6 

63.4 

7 

7 

59.7 

32 

8 

72.8 

19 

9 

68.9 

18 

10 

57.6 

9 

TOTAL 

59.3 

124 

KRUSKAL-WALLIS  STATISTIC  6.1496 

P-VALUE,  USING  CHI-SQUARED  APPROXIMATION  0.6305 

PARAMETRIC  AOV  APPLIED  TO  RANKS 


SOURCE 

DF 

SS 

MS 

F  P 

BETWEEN 

8 

7992.81 

999.102 

0.76  0.6429 

WITHIN 

115 

1 .519E+05 

1320.63 

TOTAL 

123 

1.599E+05 

TOTAL  NUMBER  OF  VALUES  THAT  WERE  TIED  114 
MAX.  DIFF.  ALLOWED  BETWEEN  TIES  0.00001 

CASES  INCLUDED  124  MISSING  CASES  13 
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KRUSKAL- WALLIS  ONE-WAY  NONPARAMETRIC  AOV  FOR  QAVG  BY  FD 


MEAN 

SAMPLE 

FD 

RANK 

SIZE 

1 

50.5 

26 

2 

87.3 

5 

3 

51.1 

10 

4 

44.2 

5 

6 

70.1 

7 

7 

68.2 

34 

8 

81.8 

22 

9 

85.8 

19 

10 

81.3 

9 

TOTAL 

68.9 

137 

KRUSKAL-WALLIS  STATISTIC  17.2974 

P-VALUE,  USING  CHI-SQUARED  APPROXIMATION  0.0272 

PARAMETRIC  AOV  APPLIED  TO  RANKS 


SOURCE 

DF 

SS 

MS 

F  P 

BETWEEN 

8 

27223.3 

3402.91 

2.33  0.0227 

WITHIN 

128 

1.868E+05 

1459.51 

TOTAL 

136 

2.140E+05 

TOTAL  NUMBER  OF  VALUES  THAT  WERE  TIED  106 
MAX.  DIFF.  ALLOWED  BETWEEN  TIES  0.00001 

CASES  INCLUDED  137  MISSING  CASES  0 
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ONE-WAY  AOV  FOR  QAVG  BY  FD 

SOURCE  DF  SS  MS  F  P 


BETWEEN  8  1123.42 
WITHIN  128  7309.46 
TOTAL  136  8432.88 


140.427  2.46  0.0165 

57.1052 


CHI-SQ 

BARTLETT'S  TEST  OF  - 

EQUAL  VARIANCES  6 . 65 


DF  P 

8  0.5753 


COCHRAN'S  Q  0.2119 

LARGEST  VAR  /  SMALLEST  VAR  4.1562 


COMPONENT  OF  VARIANCE  FOR  BETWEEN  GROUPS  5.77544 
EFFECTIVE  CELL  SIZE  14.4 


FD 

MEAN 

SAMPLE 

SIZE 

GROUP 
STD  DEV 

1 

79.115 

26 

8.3957 

2 

86.466 

5 

5.0913 

3 

78.633 

10 

10.379 

4 

78.133 

5 

7.9183 

6 

83.476 

7 

6.3270 

7 

83.078 

34 

6.8416 

8 

85.772 

22 

6.1595 

9 

85.947 

19 

8.4028 

10 

84.703 

9 

6.8057 

TOTAL 

82.814 

137 

7.5568 

CASES  INCLUDED  137  MISSING  CASES  0 


TUKEY  (HSD) 

PAIRWISE 

COMPARISONS 

OF  MEANS 

OF  QAVG 

BY  FD 

FD 

MEAN 

2 

9 

8 

10 

6 

2 

86.466 

. 

9 

85.947 

0.19 

8 

85.772 

0.26 

0.10 

10 

84.703 

0.59 

0.58 

0.51 

6 

83.476 

0.96 

1.05 

0.99 

0.46 

7 

83.078 

1.32 

1.87 

1.84 

0.81 

0.18 

1 

79.115 

2.82 

4.24* 

4.30* 

2.70 

1.92 

3 

78.633 

2.68 

3.50 

3.50 

2.47 

1.84 

4 

78.133 

2.47 

2.91 

2.89 

2.20 

1.71 

FD 

MEAN 

7 

1 

3 

7 

83.078 

1 

79.115 

2.85 

3 

78.633 

2.31 

0.24 

4 

78.133 

1.93 

0.38 

0.17 

CRITICAL  Q  VALUE  4.037  REJECTION  LEVEL  0.100 
STANDARD  ERRORS  AND  CRITICAL  VALUES  OF  DIFFERENCES 
VARY  BETWEEN  COMPARISONS  BECAUSE  OF  UNEQUAL  SAMPLE  SIZES. 
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ONE-WAY  AOV  FOR  QAVG  BY  FD 


SOURCE 

DF 

SS 

MS  F 

P 

BETWEEN 

9 

1550.53 

172.281  3.14 

0.0019 

WITHIN 

127 

6961.13 

54.8120 

TOTAL 

136 

8511.66 

CHI-SQ 

DF  P 

BARTLETT 

'S  TEST  OF  - 

-  - 

EQUAL 

VARIANCES  14.00 

9  0.1222 

COCHRAN' 

S  Q 

0.2334 

LARGEST 

VAR  / 

SMALLEST  VAR 

5.6149 

COMPONENT  OF  VARIANCE  FOR  BETWEEN  GROUPS  8.99624 
EFFECTIVE  CELL  SIZE  13.1 


FD 

MEAN 

SAMPLE 

SIZE 

GROUP 
STD  DEV 

1 

81.555 

18 

5.0111 

2 

86.466 

5 

5.0913 

3 

78.633 

10 

10.379 

4 

78.133 

5 

7.9183 

6 

83.476 

7 

6.3270 

7 

83.078 

34 

6.8416 

8 

85.772 

22 

6.1595 

9 

85.947 

19 

8.4028 

10 

84.703 

9 

6.8057 

11 

73.625 

8 

11.874 

TOTAL 

82.139 

137 

7.4035 

CASES  INCLUDED  137  MISSING  CASES  0 


TUKEY  (HSD) 

PAIRWISE 

COMPARISONS 

OF  MEANS 

OF  QAVG  BY 

FD 

FD 

MEAN 

2 

9 

8 

10 

6 

2 

86.466 

• 

9 

85.947 

0.20 

8 

85.772 

0.27 

0.11 

10 

84.703 

0.60 

0.59 

0.52 

6 

83.476 

0.98 

1.07 

1.01 

0.47 

7 

83.078 

1.35 

1.91 

1.88 

0.83 

0.18 

1 

81.555 

1.86 

2.55 

2.53 

1.47 

0.82 

3 

78.633 

2.73 

3.58 

3.58 

2.52 

1.88 

4 

78.133 

2.52 

2.97 

2.95 

2.25 

1.74 

11 

73.625 

4.30* 

5.58* 

5.62* 

4.36* 

3.64 

FD 

MEAN 

7 

1 

3 

4 

7 

83.078 

1 

81.555 

1.00 

3 

78.633 

2.36 

1.42 

4 

78.133 

1.97 

1.29 

0.17 

11 

73.625 

4.60* 

3.57 

2.02 

1.51 

CRITICAL  Q  VALUE  4.129  REJECTION  LEVEL  0.100 
STANDARD  ERRORS  AND  CRITICAL  VALUES  OF  DIFFERENCES 
VARY  BETWEEN  COMPARISONS  BECAUSE  OF  UNEQUAL  SAMPLE  SIZES. 
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KRUSKAL-WALLIS  ONE-WAY  NONPARAMETRIC  AOV  FOR  DELTA  BY  FD 


MEAN 

SAMPLE 

FD 

RANK 

SIZE 

1 

49.7 

24 

2 

92.8 

3 

3 

68.9 

9 

4 

47.8 

3 

6 

66.4 

7 

7 

66.2 

32 

8 

61.3 

19 

9 

64.6 

18 

10 

67.1 

9 

TOTAL 

65.0 

124 

KRUSKAL-WALLIS  STATISTIC  7.1940 

P-VALUE,  USING  CHI-SQUARED  APPROXIMATION  0.5159 

PARAMETRIC  AOV  APPLIED  TO  RANKS 


SOURCE 

DF 

SS 

MS 

F  P 

BETWEEN 

8 

9334.58 

1166.82 

0.89  0.5257 

WITHIN 

115 

1.503E+05 

1306.64 

TOTAL 

123 

1.596E+05 

TOTAL  NUMBER  OF  VALUES  THAT  WERE  TIED  73 
MAX.  DIFF.  ALLOWED  BETWEEN  TIES  0.00001 

CASES  INCLUDED  124  MISSING  CASES  13 
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ONE-WAY  AOV  FOR  DELTA  BY  FD 


SOURCE  DF  SS  MS  F  P 


BETWEEN  8  940.798 
WITHIN  115  14127.7 
TOTAL  123  15068.5 


117.599  0.96  0.4736 

122.849 


CHI-SQ 

BARTLETT'S  TEST  OF  - 

EQUAL  VARIANCES  11.58 


DF  P 

8  0.1710 


COCHRAN'S  Q  0.2527 

LARGEST  VAR  /  SMALLEST  VAR  70.109 


COMPONENT  OF  VARIANCE  FOR  BETWEEN  GROUPS  -0.40489 
EFFECTIVE  CELL  SIZE  13.0 


FD 

MEAN 

SAMPLE 

SIZE 

GROUP 
STD  DEV 

1 

36.513 

24 

12.651 

2 

48.444 

3 

1.8954 

3 

42.148 

9 

6.8416 

4 

36.111 

3 

10.205 

6 

41.761 

7 

10.260 

7 

42.437 

32 

10.662 

8 

40.473 

19 

9.5415 

9 

41.666 

18 

10.972 

10 

44.037 

9 

15.870 

TOTAL 

41.510 

124 

11.083 

CASES  INCLUDED  124  MISSING  CASES  13 


TUKEY  (HSD)  PAIRWISE  COMPARISONS  OF  MEANS  OF  DELTA  BY  FD 

HOMOGENEOUS 


FD 

MEAN 

GROUPS 

2 

48.444 

I 

10 

44.037 

I 

7 

42.437 

I 

3 

42.148 

I 

6 

41.761 

I 

9 

41.666 

I 

8 

40.473 

I 

1 

36.513 

I 

4 

36.111 

I 

THERE  ARE  NO  SIGNIFICANT  PAIRWISE  DIFFERENCES  AMONG  THE  MEANS. 

CRITICAL  Q  VALUE  4.099  REJECTION  LEVEL  0.100 
STANDARD  ERRORS  AND  CRITICAL  VALUES  OF  DIFFERENCES 
VARY  BETWEEN  COMPARISONS  BECAUSE  OF  UNEQUAL  SAMPLE  SIZES. 
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ONE-WAY  AOV  FOR  DELTA  BY  FD 


SOURCE  DF  SS  MS  F  P 


BETWEEN  9  933.124 
WITHIN  114  14096.5 
TOTAL  123  15029.6 


103.680 

123.654 


0.84  0.5833 


CHI-SQ 

BARTLETT'S  TEST  OF  - 

EQUAL  VARIANCES  12.37 


DF  P 

9  0.1930 


COCHRAN'S  Q  0.2098 

LARGEST  VAR  /  SMALLEST  VAR  70.109 


COMPONENT  OF  VARIANCE  FOR  BETWEEN  GROUPS  -1.69922 
EFFECTIVE  CELL  SIZE  11.8 


FD 

MEAN 

SAMPLE 

SIZE 

GROUP 
STD  DEV 

1 

35.708 

16 

11.740 

2 

48.444 

3 

1.8954 

3 

42.148 

9 

6.8416 

4 

36.111 

3 

10.205 

6 

41.761 

7 

10.260 

7 

42.437 

32 

10.662 

8 

40.473 

19 

9.5415 

• 

41.666 

18 

10.972 

1C 

44.037 

9 

15.870 

1  i 

38.125 

8 

15.036 

TOTAL 

41.091 

124 

11.119 

CASES  INCLUDED 

124 

MISSING 

CASES  13 

TUKEY  (HSD)  PAIRWISE  COMPARISONS  OF  MEANS  OF  DELTA  BY  FD 

HOMOGENEOUS 


-D 

MEAN 

GROUPS 

2 

48.444 

I 

10 

44.037 

I 

7 

42.437 

I 

3 

42.148 

I 

6 

41.761 

I 

9 

41.666 

I 

8 

40.473 

I 

11 

38.125 

I 

4 

36.111 

I 

1 

35.708 

I 

THERE  ARE  NO  SIGNIFICANT  PAIRWISE  DIFFERENCES  AMONG  THE  MEANS. 

CRITICAL  Q  VALUE  4.195  REJECTION  LEVEL  0.100 
STANDARD  ERRORS  AND  CRITICAL  VALUES  OF  DIFFERENCES 
VARY  BETWEEN  COMPARISONS  BECAUSE  OF  UNEQUAL  SAMPLE  SIZES. 
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KRUSKAL-WALLIS  ONE-WAY  NONPARAMETRIC  AOV  FOR  PRETOT  BY  FD 


MEAN 

SAMPLE 

FD 

RANK 

SIZE 

1 

49.6 

18 

2 

30.8 

3 

3 

64.4 

6 

4 

55.2 

5 

6 

48.5 

4 

7 

64.3 

33 

8 

53.3 

16 

9 

52.9 

17 

10 

39.5 

7 

TOTAL 

51.0 

109 

KRUSKAL-WALLIS  STATISTIC 

P-VALUE,  USING  CHI-SQUARED  APPROXIMATION 
PARAMETRIC  AOV  APPLIED  TO  RANKS 


SOURCE 

DF 

SS 

MS 

F 

BETWEEN 

8 

9420.25 

1177.53 

1.17 

WITHIN 

100 

1.003E+05 

1002.51 

TOTAL 

108 

1.097E+05 

TOTAL  NUMBER  OF  VALUES  THAT  WERE  TIED  35 
MAX.  DIFF.  ALLOWED  BETWEEN  TIES  0.00001 

CASES  INCLUDED  109  MISSING  CASES  0 


9.2767 

0.3195 


P 


0.3217 
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KRUSKAL-WALLIS  ONE-WAY  NONPARAMETRIC  AOV  FOR  POSTTOT  BY  FD 


MEAN 

SAMPLE 

FD 

RANK 

SIZE 

1 

41.6 

18 

2 

20.8 

3 

3 

59.8 

6 

4 

47.2 

5 

6 

53.1 

4 

7 

70.1 

33 

8 

54.3 

16 

9 

54.4 

17 

10 

38.4 

7 

TOTAL 

48.9 

109 

KRUSKAL-WALLIS  STATISTIC  20.0436 

P-VALUE,  USING  CHI-SQUARED  APPROXIMATION  0.0102 

PARAMETRIC  AOV  APPLIED  TO  RANKS 

SOURCE  DF  SS  MS  F  P 


BETWEEN  8  20783.9  2597.99  2.85  0.0069 

WITHIN  100  91205.3  912.053 

TOTAL  108  1.120E+05 


TOTAL  NUMBER  OF  VALUES  THAT  WERE  TIED  47 
MAX.  DIFF.  ALLOWED  BETWEEN  TIES  0.00001 

CASES  INCLUDED  109  MISSING  CASES  0 
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ONE-WAY  AOV  FOR  POSTTOT  BY  FD 


SOURCE 

DF 

SS 

MS 

F  P 

BETWEEN 

8 

53674.4 

6709.30 

2.27  0.0280 

WITHIN 

100 

2.951E+05 

2950.72 

TOTAL 

108 

3.487E+05 

CHI-SQ  DF  P 

BARTLETT'S  TEST  OF  -  -  - 

EQUAL  VARIANCES  9.47  8  0.3046 

COCHRAN'S  Q  0.2591 

LARGEST  VAR  /  SMALLEST  VAR  6.5061 

COMPONENT  OF  VARIANCE  FOR  BETWEEN  GROUPS  334.847 
EFFECTIVE  CELL  SIZE  11.2 


SAMPLE  GROUP 


FD 

MEAN 

SIZE 

STD  DEV 

1 

211.27 

18 

53.753 

2 

185.66 

3 

30.038 

3 

248.83 

6 

43.254 

4 

227.80 

5 

34.881 

6 

233.75 

4 

53.500 

7 

261.30 

33 

46.013 

8 

232.62 

16 

57.834 

9 

228.70 

17 

76.620 

10 

212.28 

7 

39.756 

TOTAL 

226.91 

109 

54.320 

CASES  INCLUDED 

109 

MISSING 

CASES  0 

TUKEY  (HSD) 

PAIRWISE 

COMPARISONS 

OF  MEANS 

OF  POSTTOT 

BY  FD 

FD 

MEAN 

7 

3 

6 

8 

9 

7 

261.30 

3 

248.83 

0.73 

6 

233.75 

1.35 

0.61 

8 

232.62 

2.45 

0.88 

0.05 

9 

228.70 

2.84 

1.10 

0.24 

0.29 

4 

227.80 

1.82 

0.90 

0.23 

0.25 

0.05 

10 

212.28 

3.07 

1.71 

0.89 

1.17 

0.95 

1 

211.27 

4.44* 

2.07 

1.06 

1.62 

1.34 

2 

185.66 

3.27 

2.33 

1.64 

1.94 

1.79 

FD 

MEAN 

4 

10 

1 

4 

227.80 

10 

212.28 

0.69 

1 

211.27 

0.85 

0.06 

2 

185.66 

1.50 

1.00 

1.07 

CRITICAL  Q  VALUE  4.108  REJECTION  LEVEL  0.100 
STANDARD  ERRORS  AND  CRITICAL  VALUES  OF  DIFFERENCES 
VARY  BETWEEN  COMPARISONS  BECAUSE  OF  UNEQUAL  SAMPLE  SIZES. 
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ONE-WAY  AOV  FOR  POSTTOT  BY  FD 


SOURCE 


DF 


SS 


MS 


BETWEEN 

WITHIN 

TOTAL 


9 

99 

108 


57113.8 

2.950E+05 

3.521E+05 


6345.98 

2979.98 


2.13  0.0337 


BARTLETT'S  TEST  OF 
EQUAL  VARIANCES 


CHI-SQ 

10.08 


DF 

9 


P 

0-3441 


COCHRAN'S  Q 

LARGEST  VAR  /  SMALLEST  VAR 


0.2325 

6.5061 


COMPONENT  OF  VARIANCE  FOR  BETWEEN  GROUPS 
EFFECTIVE  CELL  SIZE 


332.465 

10.1 


FD 

MEAN 

SAMPLE 

SIZE 

GROUP 
STD  DEV 

1 

212.50 

12 

60.043 

2 

185.66 

3 

30.038 

3 

248.83 

6 

43.254 

4 

227.80 

5 

34.881 

6 

233.75 

4 

53.500 

7 

261.30 

33 

46.013 

8 

232.62 

16 

57.834 

9 

228.70 

17 

76.620 

10 

212,28 

7 

39.756 

11 

208.83 

6 

43.379 

TOTAL 

225.23 

109 

54.589 

CASES  INCLUDED 

109 

MISSING 

CASES  0 

TUKEY  (HSD) 

FD 

PAIRWISE 

MEAN 

COMPARISONS  OF  MEANS 

HOMOGENEOUS 

GROUPS 

7 

261.30 

I 

3 

248.83 

I 

6 

233.75 

I 

8 

232.62 

I 

9 

228.70 

I 

4 

227.80 

I 

1 

212.50 

I 

10 

212.28 

I 

11 

208.83 

I 

2 

185.66 

I 

OF  POSTTOT  BY  FD 


THERE  ARE  NO  SIGNIFICANT  PAIRWISE  DIFFERENCES  AMONG  THE  MEANS. 

CRITICAL  Q  VALUE  4.205  REJECTION  LEVEL  0.100 
STANDARD  ERRORS  AND  CRITICAL  VALUES  OF  DIFFERENCES 
VARY  BETWEEN  COMPARISONS  BECAUSE  OF  UNEQUAL  SAMPLE  SIZES. 
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KRUSKAL-WALLIS  ONE-WAY  NONPARAMETRIC  AOV  FOR  DELTA  BY  FD 


MEAN 

SAMPLE 

FD 

RANK 

SIZE 

1 

49.2 

18 

2 

50.5 

3 

3 

54.5 

6 

4 

45.9 

5 

6 

59.6 

4 

7 

60.1 

33 

8 

53.3 

16 

9 

52.6 

17 

10 

61.9 

7 

TOTAL 

54.2 

109 

KRUSKAL-WALLIS  STATISTIC  2  5544 

P-VALUE,  USING  CHI-SQUARED  APPROXIMATION  0.9591 

PARAMETRIC  AOV  APPLIED  TO  RANKS 


SOURCE 

DF 

SS 

MS 

F  P 

BETWEEN 

8 

2553.53 

319.192 

0.30  0.9626 

WITHIN 

100 

1.054E+05 

1054.10 

TOTAL 

108 

1.080E+05 

TOTAL  NUMBER  OF  VALUES  THAT  WERE  TIED  63 
MAX.  DIFF.  ALLOWED  BETWEEN  TIES  0.00001 

CASES  INCLUDED  109  MISSING  CASES  0 
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Appendix  E:  Summary  of  Supervisor  Comments 


"This  student  has  14  years  experience  with  7  of  those  in  a  SPO.  He  has  a 
Masters  from  AFIT  in  Logistics,  took  this  course  to  fill  the  square,  and  yet 
still  got  something  out  of  the  course.  Also  this  employee  is  my  top  program 
manager." 

"Student  was  an  exeptional  performer  before  and  after  SYS200.  The  course 
served  as  a  square  filler  and  an  updater  on  the  latest  acronyms,  but  this 
student  knew  the  subject  matter  well,  long  before  attending.  People  like  this 
should  be  able  to  CLEP  the  course-would  save  TDY  fimds  and  better  use 
their  time.  This  is  not." 

"The  benefit  of  SYS200  to  someone  who  has  been  in  the  field  for  many  years 
is  minimal.  There  should  be  greater  latitude  to  qualify  out  of  these  often 
"block  filling"  requirements." 

[Student  in  Comm/Computer  Functional  Discipline] 

"Need  more  emphasis  on  Software  acquisition/support  and  its  relationship  to 
systems  engineering  processes." 

[Student  in  S&T/Engineering  Functional  Discipline] 

"Ms, _ is  a  senior  Air  Force  executive  (GM-15)  with  several  years 

experience  as  a  system  program  manager.  She  possesses  a  tremendous 
quantitative  backgroimd  -  degrees  in  math  and  operations  research.  Systems 
200  is  an  excellent  course  for  the  majority  of  those  attending  -  not  an 

intellectual  challenge  for  Ms. _ ." 

[Student  in  Acquisition  Logistics  Functional  Disciphne] 

"[student]  tells  me  that  use  of  video  camera  inhibited  student  participation. 
Video  negated  non-attribution  so  student  reluctant  to  tell  valuable  lessons 
learned  regarding  actual  program  experiences." 

[Student  in  Project  Management  Functional  Discipline] 

"Capt. _ was  already  good!  SYS200  gave  him  some  polish!  Keep  doing 

what  you're  doing!" 

[Student  in  Project  Management  Fimctional  Discipline] 

"Smaller  class  would  have  provided  a  better  interchange." 

[Student  in  Comm/Computers  Functional  Discipline] 
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"The  individual  is  putting  a  program  together  in  preparation  of  source 
selection.  The  team  goals  helped  to  prepare  him  for  the  challenge  and  added 
to  the  knowledge  he  has." 

[Student  in  Acquisition  Logistics  Functional  Discipline] 

"It  is  too  early  to  accurately  assess  tangible  job  performance  improvements 

related  to  Capt. _ 's  completion  of  SYS200.  I  am  sure  that  improvement  will 

be  noted;  but  accurate  assessments  as  requested  are  premature." 

[Student  in  Acquisition  Logistics  Functional  Discipline] 

"Stop  using  volimteer  guest  speakers,  train  dedicated  instructors.  This  is  an 
important  coiu^e." 

[Student  in  Acquisition  Logistics  Functional  Discipline] 

"Three  weeks  is  MUCH  TOO  LONG  for  people  in  this  laboratory  to  attend 
this  type  of  training.  One  week  should  suffice  with  the  exception  of  those 
who  are  directly  responsible  for  contractual  efforts." 

[Student  in  S&T/Engineering  Functional  Discipline]. 

"Increase  Availability." 

[Student  in  Financial  Management  Fimctional  Discipline] 

"Would  like  to  see  some  modules  taught  on  COTS  acquisition.  Overall  the 
course  is  excellent.  Do  not  lessen  the  course  in  scope  or  work  expected  of  the 
student." 

[Student  in  Acquisition  Logistics  Functional  Discipline] 

"[student]  has  a  better  understanding  of  the  acquisition  process,  but  our  day 
to  day  work  does  not  require  overall  program  visibility.  The  best  part  of  the 
course  for  engineers  working  at  test  centers  is  the  introduction  of  the  overall 
acquisition  cycle,  the  chain  of  command  or  management,  and  imderstanding 
the  acronyms  used,  [student]  has  told  me  that  the  course  was  well  taught 
and  all  his  questions  were  answered." 

[Student  in  Test  &  Evaluation  Functional  Discipline] 

"SAF/AQ  PEM  job  -  doesn't  have  the  chance  to  directly  apply  what  he 
learned,  but  it  is  very  important  nevertheless,  since  he  directs  SPO's  & 
interfaces  with  OSD  and  Congress." 

"It  is  difficult  to  determine  course  impact  within  two  months." 
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"SYS200  doesn’t  change  anyone  overnight  -  haven't  seen  drastic  changes.  I 
feel  his  ability  to  manage  in  the  future  will  be  enhanced." 

"Already  a  top  performer." 

"Course  is  just  a  square  filler  for  certification." 

"Make  course  shorter  -  eliminate  team  exercises." 

Number  of  Responses  With  No  Comment  -  39 
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